{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import numpy as np\n",
    "import os\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Variational Autoencoder in MXNet\n",
    "\n",
    "#### Xiaoyu Lu,  July 5th, 2017\n",
    "\n",
    "This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "To complete this tutorial, we need following python packages:\n",
    "\n",
    "- numpy, matplotlib "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading the Data\n",
    "\n",
    "We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mnist = mx.test_utils.get_mnist()\n",
    "image = np.reshape(mnist['train_data'],(60000,28*28))\n",
    "label = image\n",
    "image_test = np.reshape(mnist['test_data'],(10000,28*28))\n",
    "label_test = image_test\n",
    "[N,features] = np.shape(image)          #number of examples and features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADFCAYAAACxSv92AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFzNJREFUeJzt3Xuw1XW5x/HPg8rdGMEtoYfj9gJqOZyNbdBGOYOCRFp5\noRwtHUpHgkzFQHFwTNMsrGMKURYIgdnxWIi3xjlSdPIymYlhBwEvpRBX2YxQQolcnvMH68wQ6/nF\nWnvd9lrf92umYe/P/u7fen6sZ+8efq7fd5m7CwAAAEhRp1oXAAAAANQKwzAAAACSxTAMAACAZDEM\nAwAAIFkMwwAAAEgWwzAAAACSxTAMAACAZDEMAwAAIFkMwwAAAEjWwaV8s5mNljRd0kGS7nP3af9s\n/eGHH+7Nzc2lPCSgl156abO7N1XzMeldlGrVqlXavHmzVfMx6VuUA79zUa8K7d12D8NmdpCk70k6\nW9JaSS+a2ePuviLre5qbm7VkyZL2PiQgSTKz1dV+THoXpWptba36Y9K3KAd+56JeFdq7pbxMYqik\nP7r7m+7+vqT/knReCccDAAAAqqqUYfgoSWv2+XxtLvsHZjbOzJaY2ZK2trYSHg6oLnoX9Yi+Rb2i\nd1ErFb+Bzt1nuXuru7c2NVX1JUdASehd1CP6FvWK3kWtlDIMr5PUf5/P/yWXAQAAAHWhlGH4RUkD\nzOwYM+ss6WJJj5enLAAAAKDy2r2bhLvvMrMvS3pKe7dWm+vuy8tWGQAAAFBhJe0z7O5PSnqyTLUA\nAAAAVcU70AEAACBZDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZDMMAAABIFsMwAAAAksUwDAAA\ngGQxDAMAACBZDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZB9e6AACNbc2aNWE+ffr0ML/77rvz\nsuuuuy5ce+2114Z5//79C6wOAJA6rgwDAAAgWQzDAAAASBbDMAAAAJLFMAwAAIBkMQwDAAAgWewm\nUWF79uwJ8x07dpTl+PPnzw/z7du352UrVqwI195zzz1hPnXq1DCfOXNmmHfr1i3M77rrrrxswoQJ\n4VrUr3Xr1oX54MGDw3zr1q1hbmZ5WVaPZvV/W1tbmAMd2cqVK8N85MiRYf7yyy+HeVNTU9lqQppm\nz54d5uPHjw/zrFnntddeC/OBAwe2r7AKKWkYNrNVkt6VtFvSLndvLUdRAAAAQDWU48rwme6+uQzH\nAQAAAKqK1wwDAAAgWaUOwy7pl2b2kpmNixaY2TgzW2JmS3gdH+oJvYt6RN+iXtG7qJVSh+Ez3L1F\n0sclXWVm/77/Anef5e6t7t7Ki/pRT+hd1CP6FvWK3kWtlPSaYXdfl/tzk5k9ImmopGfKUVg1/eUv\nfwnz3bt3h/kf/vCHvGzRokXh2qw75mfNmlVgdeXT3Nwc5pMmTQrzOXPmhHmvXr3CfNiwYWF+1lln\nHbg41I3Vq1eH+fDhw8N8y5YtYR7tGiHF/dWlS5dw7aZNm8L8zTffDPOjjz46zA866KAwT9kbb7yR\nl2U9l0OHDq10OUl44YUXwnzEiBFVrgQpWbx4cV72la98JVzbqVNx11Czfs93NO2+MmxmPczs0P//\nWNIoSa+UqzAAAACg0kq5MtxX0iO5qf9gSf/p7v9dlqoAAACAKmj3MOzub0r6tzLWAgAAAFQVW6sB\nAAAgWQzDAAAASFY53oGubqxduzbMW1pawjzrzul6EN3xmbU7RLdu3cL8iiuuCPMjjjgizHv27Bnm\nbJHTse3cuTPMs3aNGD16dJivWbOmLPVEP4933HFHuPaMM84I8wEDBoR51i4uWb2esugO81dffTVc\ny24SxXH3MI928JCk119/vZLlIHFRf7333ns1qKR2uDIMAACAZDEMAwAAIFkMwwAAAEgWwzAAAACS\nxTAMAACAZCW1m0SfPn3CvG/fvmFei90kRo0aFeZZtS9cuDDMu3TpkpcNHz683XWhcV1//fVhPnPm\nzCpXstfTTz+dl23fvj1ce8EFF4R51s/F0qVL219YYmbMmJGXZf1+QnG2bdsW5t/85jfD/Nprrw1z\ndupBMVasWBHmt956a8HHOOWUU8J80aJFYd6jR4+Cj11LXBkGAABAshiGAQAAkCyGYQAAACSLYRgA\nAADJYhgGAABAspLaTaJbt25hPm/evDBfsGBBmH/0ox/Ny8aMGVNULWeccUaYP/bYY2HeuXPnMN+4\ncWOYT58+vah6kIY1a9bkZQ888EC41t2LOnbWzg5ZPxuXXnppmPfv3z8vO+mkk8K1U6ZMCfOsn91i\nzyllu3fvrnUJDWv8+PFFrc/qfyDyxz/+MczPOeecMH/nnXcKPva0adPCvFevXgUfoyPiyjAAAACS\nxTAMAACAZDEMAwAAIFkMwwAAAEgWwzAAAACSdcDdJMxsrqRPSNrk7ifnst6SHpLULGmVpIvcfUvl\nyqysIUOGhPmgQYPCPNrZ4YYbbgjXfutb3wrz22+/veBj/zMf/OAHwzzrPe6RhnXr1oX54MGD87Kt\nW7eGa80szD/3uc+F+ezZs8N8xYoVRa2/+OKL87Lu3buHa4888sgw79Qp/nf+j3/84zC/8cYbwzza\n2aLRrF+/PsyzegilK+bufUk6++yzK1QJGtF9990X5tFuQlkuvPDCMD/zzDPbVVNHV8iV4XmSRu+X\n3ShpsbsPkLQ49zkAAABQVw44DLv7M5L2/2fseZLm5z6eL+n8MtcFAAAAVFx7XzPc19035D7eKKlv\n1kIzG2dmS8xsSVtbWzsfDqg+ehf1iL5FvaJ3USsl30Dne9/SKfNtndx9lru3untrU1NTqQ8HVA29\ni3pE36Je0buolfa+HfPbZtbP3TeYWT9Jm8pZVEfRpUuXgtcedthhRR17xowZYT5s2LAwz7qZCWnb\nvHlzmN95551hvmVL/n2uffvG/2HnmGOOCfMJEyaEedbNny0tLUXllfS3v/0tzL/97W+HedbPaSNZ\ntGhRmGf9XaFw27dvD/Nly5YVdZw+ffqUoxw0mGJ/n2XdWBz1V9ZN/o2qvVeGH5c0NvfxWEmPlacc\nAAAAoHoOOAyb2YOSnpd0gpmtNbMrJE2TdLaZvSFpZO5zAAAAoK4c8GUS7n5JxpdGlLkWAAAAoKp4\nBzoAAAAki2EYAAAAyWrvbhLYz8SJE8P8d7/7XZg/8sgjYb58+fIwP/nkk9tXGBrCrl27wnzy5Mlh\n/sADD4R5r1698rKnnnoqXHv88ceH+c6dO8O8nr311lu1LqFmXnnllYLX1mIHkHp20003hXnWW2AP\nGjQozLN2akEatm7dGubnnXdeWY5/66235mUnnnhiWY5dL7gyDAAAgGQxDAMAACBZDMMAAABIFsMw\nAAAAksUwDAAAgGSxm0SZZN3tO2vWrDBfvHhxmGfdHXr++eeH+emnnx7mF1xwQV5mZuFadHx//vOf\nwzxr14gsv/3tb/OygQMHFnWMbt26FbUejePUU0+tdQlVs2PHjjB/6aWXwjz6Xf/QQw8V9ZgzZswI\n865duxZ1HDSWZ599Nsx/85vfFHWcz3zmM2H++c9/vtiSGg5XhgEAAJAshmEAAAAki2EYAAAAyWIY\nBgAAQLIYhgEAAJAsdpOosN69e4f5U089FeajR48O83vuuaeofO7cuXnZmDFjwrU9e/YMc3QcV111\nVZi7e5hHu4lIxe8cUa/27NkT5p06xf/+z/p7xD/aunVrRY+/fv36MI+ez6effjpc+9Zbb4X5+++/\nH+bf/e53w3z37t1h3qNHjzAfNWpUXpa1C8TOnTvD/KSTTgpzpOHFF18M87FjxxZ1nE9+8pNhPnv2\n7DBntxKuDAMAACBhDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZB9xNwszmSvqEpE3ufnIuu1XS\nlZLacsumuvuTlSqyEQ0dOjTMly9fHubXXXddmP/sZz8L88svvzwv+9Of/hSuvf7668P80EMPDXNU\nztKlS8P8mWeeCXMzC/Os96BPRdauEVl/X62trZUsp0Pr3r17mEd/V5/61KfCtSeccEJZann++efD\nPNrt4+CD4//7ytod59RTTw3zyZMnh/mwYcPCvKWlJcyjXSb69+8frt2+fXuYNzU1hTkaS9auLKed\ndlpZjn/88ceHedZOKCjsyvA8SdF+X3e7e0vufwzCAAAAqDsHHIbd/RlJ71ShFgAAAKCqSnnN8NVm\n9r9mNtfMDstaZGbjzGyJmS1pa2vLWgZ0OPQu6hF9i3pF76JW2jsM3yvpWEktkjZIuitrobvPcvdW\nd2/l9VCoJ/Qu6hF9i3pF76JW2jUMu/vb7r7b3fdImi0pvhsMAAAA6MAOuJtExMz6ufuG3KcXSHql\nfCWlrV+/fmE+b968MB8/fnyYjxw5Mi+74447wrWvvfZamD/00ENhjsp57733wnzHjh1hfuSRR4b5\nueeeW7aaOoJdu3aF+YwZM4o6zqc//ekwnzp1atE1NYrbbrstzI877ri87Ne//nVFaxkwYECYf/az\nn83Lsu6YP+aYY8paU6GefDL/PvKNGzeGa0888cRKl4MO7K674v+YnrULTrGmTJlSluOkpJCt1R6U\nNFzS4Wa2VtItkoabWYskl7RK0hcrWCMAAABQEQccht39kiCeU4FaAAAAgKriHegAAACQLIZhAAAA\nJIthGAAAAMlq124SqL6uXbuG+fDhw8P8oIMOysuy7sh/9NFHwzxrl4kTTjghzFF9WX3Rs2fPKldS\nPlGf3nvvveHaG264Icybm5vD/Kabbgrzzp07F1ZcQsaOHVtQhr1+/vOfF7z28ssvr2Al6EjWrVuX\nly1YsKAsx/7CF74Q5uzRXDyuDAMAACBZDMMAAABIFsMwAAAAksUwDAAAgGRxA10Hs379+jBfuHBh\nmD///PNhnnWzXGTIkCFhPnDgwIKPgdq47LLLal1Cu0U3lkjSnXfemZd9//vfD9dm3UAye/bs9hcG\nVNiFF15Y6xJQJa2trXnZ5s2bizrGxz72sTCfOXNmu2pCPq4MAwAAIFkMwwAAAEgWwzAAAACSxTAM\nAACAZDEMAwAAIFnsJlFhbW1tYf69730vzH/0ox+F+dq1a0uuJXqLZin7rWvNrOTHRHHcvah83rx5\nYX7zzTeXq6SSPfjgg2F+9dVXh/mWLVvysmuuuSZce/fdd7e/MACosE2bNuVlnToVdx1yypQpYc7b\nyJcPV4YBAACQLIZhAAAAJIthGAAAAMliGAYAAECyGIYBAACQrAPuJmFm/SXdL6mvJJc0y92nm1lv\nSQ9Japa0StJF7p5/G3gD2rZtW172xBNPhGtvu+22MH/99dfLWtP+zjrrrLxs2rRp4dqPfOQjFa0F\nhcvawSMrz9plJKvvrrjiirzs0EMPDdcuX748zH/4wx+G+bPPPhvmq1atCvPjjjsuzC+++OK8LGs3\nCaAjy9oFZvXq1WF+7LHHVrIcVNDkyZPDfM+ePSUfe9CgQSUfA/9cIVeGd0ma5O4fknSapKvM7EOS\nbpS02N0HSFqc+xwAAACoGwccht19g7v/Pvfxu5JWSjpK0nmS5ueWzZd0fqWKBAAAACqhqNcMm1mz\npMGSXpDU19035L60UXtfRhF9zzgzW2JmS7LegALoiOhd1CP6FvWK3kWtFDwMm1lPSQ9Lmujuf933\na773hVHhi6PcfZa7t7p7a1NTU0nFAtVE76Ie0beoV/QuaqWgYdjMDtHeQfgn7r4wF79tZv1yX+8n\nKf89BwEAAIAOrJDdJEzSHEkr3f07+3zpcUljJU3L/flYRSqsgu3bt4f5mjVrwvzSSy/Ny5YuXVrW\nmvY3atSoMP/a174W5kOGDMnLsnYkQP3avXt3mGftJjFnzpy8rHfv3uHaZcuWtb+wfXz84x8P89Gj\nR4f5l7/85bI8LlBrWb9zy7HDAGpj3bp1Yb5gwYIw79Qp/5pjly5dwrW33HJLmPfo0aPA6tBeBxyG\nJZ0u6TJJy8zs5Vw2VXuH4J+a2RWSVku6qDIlAgAAAJVxwGHY3Z+TlHVJcUR5ywEAAACqh3egAwAA\nQLIYhgEAAJAshmEAAAAkq5Ab6OrO3//+9zCfOHFimD/33HNh/uqrr5atpv2dc845Yf7Vr341zFta\nWsL8kEMOKVtNqL0Pf/jDYT5y5Mgw/+Uvf1nU8deuXZuXZd0dneWII44I8wkTJoT5zTffXNTxgUb3\nq1/9KsxHjOA2nI5u27ZtYV7M79Hm5uYwnzJlSntKQhlwZRgAAADJYhgGAABAshiGAQAAkCyGYQAA\nACSLYRgAAADJqpvdJFatWpWXfeMb3wjXZt1hv3r16nKW9A+6d+8e5rfffnuYf+lLXwrzzp07l60m\n1J8PfOADYZ71vvf3339/mF9zzTUl1/L1r389zK+88sow79OnT8mPCTQSd691CQAKwJVhAAAAJIth\nGAAAAMliGAYAAECyGIYBAACQLIZhAAAAJKtudpN4+OGH87I5c+aU5dinnHJKmF9yySVhfvDB+X9t\n48aNC9d27dq1/YUBOT179gzzrF1JsnIAlTFmzJi87Ac/+EENKkElHXXUUWF+7rnnhvkTTzxRyXJQ\nJlwZBgAAQLIYhgEAAJAshmEAAAAki2EYAAAAyTrgDXRm1l/S/ZL6SnJJs9x9upndKulKSW25pVPd\n/clKFTpp0qSCMgAAqm3EiBF52Z49e2pQCSop62bmRx99tMqVoJwK2U1il6RJ7v57MztU0ktm9ovc\n1+529/+oXHkAAABA5RxwGHb3DZI25D5+18xWSor3FgEAAADqSFGvGTazZkmDJb2Qi642s/81s7lm\ndljG94wzsyVmtqStrS1aAnRI9C7qEX2LekXvolYKHobNrKekhyVNdPe/SrpX0rGSWrT3yvFd0fe5\n+yx3b3X31qampjKUDFQHvYt6RN+iXtG7qJWChmEzO0R7B+GfuPtCSXL3t919t7vvkTRb0tDKlQkA\nAACU3wGHYTMzSXMkrXT37+yT99tn2QWSXil/eQAAAEDlFLKbxOmSLpO0zMxezmVTJV1iZi3au93a\nKklfrEiFAAAAQIUUspvEc5Is+FLF9hQGAAAAqoF3oAMAAECyGIYBAACQLIZhAAAAJIthGAAAAMli\nGAYAAECyGIYBAACQLIZhAAAAJIthGAAAAMliGAYAAECyzN2r92BmbZJWV+0BK+9wSZtrXUSVdKRz\nPdrdm6r5gPRu3epI50nflkdHek4rqSOdJ71buo70fFZaRzrXgnq3qsNwozGzJe7eWus6qiGlc01B\nKs9nKueZklSe01TOMxUpPZ/1eK68TAIAAADJYhgGAABAshiGSzOr1gVUUUrnmoJUns9UzjMlqTyn\nqZxnKlJ6PuvuXHnNMAAAAJLFlWEAAAAki2EYAAAAyWIYLpCZzTWzTWb2yj5ZbzP7hZm9kfvzsFrW\nWA5m1t/M/sfMVpjZcjO7Npc33Lmmgt5tvHNNAX3beOeaCnq3/s6VYbhw8ySN3i+7UdJidx8gaXHu\n83q3S9Ikd/+QpNMkXWVmH1Jjnmsq5onebbRzTcE80beNdq6pmCd6t67OlWG4QO7+jKR39ovPkzQ/\n9/F8SedXtagKcPcN7v773MfvSlop6Sg14Lmmgt5tvHNNAX3beOeaCnq3/s6VYbg0fd19Q+7jjZL6\n1rKYcjOzZkmDJb2gBj/XBDX080nvNqyGfi7p24bW0M9nvfcuw3CZ+N496hpmnzoz6ynpYUkT3f2v\n+36t0c41dY32fNK7aWi055K+TUejPZ+N0LsMw6V528z6SVLuz001rqcszOwQ7W3sn7j7wlzckOea\nsIZ8PundhteQzyV9m4SGfD4bpXcZhkvzuKSxuY/HSnqshrWUhZmZpDmSVrr7d/b5UsOda+Ia7vmk\nd5PQcM8lfZuMhns+G6l3eQe6ApnZg5KGSzpc0tuSbpH0qKSfSvpXSaslXeTu+79ovq6Y2RmSnpW0\nTNKeXDxVe18H1FDnmgp6l96tR/QtfVuv6N36612GYQAAACSLl0kAAAAgWQzDAAAASBbDMAAAAJLF\nMAwAAIBkMQwDAAAgWQzDAAAASBbDMAAAAJL1f1iM7ql26a6bAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11234e9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, (ax1, ax2, ax3, ax4) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))\n",
    "ax1.imshow(np.reshape(image[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax2.imshow(np.reshape(image[1,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax3.imshow(np.reshape(image[2,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax4.imshow(np.reshape(image[3,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_prefix = None\n",
    "\n",
    "batch_size = 100\n",
    "nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},\n",
    "                            batch_size = batch_size)\n",
    "nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},\n",
    "                            batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.  Building the Network Architecture\n",
    "\n",
    "### 2.1 Gaussian MLP as encoder\n",
    "Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:\n",
    "\n",
    "\\begin{align}\n",
    "\\log q_{\\phi}(z|x) &= \\log \\mathcal{N}(z:\\mu,\\sigma^2I) \\\\\n",
    "\\textit{ where } \\mu &= W_2h+b_2, \\log \\sigma^2 = W_3h+b_3\\\\\n",
    "h &= \\tanh(W_1x+b_1)\n",
    "\\end{align}\n",
    "\n",
    "where $\\{W_1,W_2,W_3,b_1,b_2,b_3\\}$ are the weights and biases of the MLP.\n",
    "Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## define data and loss labels as symbols \n",
    "data = mx.sym.var('data')\n",
    "loss_label = mx.sym.var('loss_label')\n",
    "\n",
    "## define fully connected and activation layers for the encoder, where we used tanh activation function.\n",
    "encoder_h  = mx.sym.FullyConnected(data=data, name=\"encoder_h\",num_hidden=400)\n",
    "act_h = mx.sym.Activation(data=encoder_h, act_type=\"tanh\",name=\"activation_h\")\n",
    "\n",
    "## define mu and log variance which are the fully connected layers of the previous activation layer\n",
    "mu  = mx.sym.FullyConnected(data=act_h, name=\"mu\",num_hidden = 5)\n",
    "logvar  = mx.sym.FullyConnected(data=act_h, name=\"logvar\",num_hidden = 5)\n",
    "\n",
    "## sample the latent variables z according to Normal(mu,var)\n",
    "z = mu + np.multiply(mx.symbol.exp(0.5 * logvar), \n",
    "                     mx.symbol.random_normal(loc=0, scale=1, shape=np.shape(logvar.get_internals()[\"logvar_output\"])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Bernoulli MLP as decoder\n",
    "\n",
    "In this case let $p_\\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:\n",
    "\n",
    "\\begin{align}\n",
    "\\log p(x|z) &= \\sum_{i=1}^D x_i\\log y_i + (1-x_i)\\log (1-y_i) \\\\\n",
    "\\textit{ where }  y &= f_\\sigma(W_5\\tanh (W_4z+b_4)+b_5)\n",
    "\\end{align}\n",
    "\n",
    "where $f_\\sigma(\\dot)$ is the elementwise sigmoid activation function, $\\{W_4,W_5,b_4,b_5\\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define fully connected and tanh activation layers for the decoder\n",
    "decoder_z = mx.sym.FullyConnected(data=z, name=\"decoder_z\",num_hidden=400)\n",
    "act_z = mx.sym.Activation(data=decoder_z, act_type=\"tanh\",name=\"activation_z\")\n",
    "\n",
    "# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension\n",
    "decoder_x = mx.sym.FullyConnected(data=act_z, name=\"decoder_x\",num_hidden=features)\n",
    "y = mx.sym.Activation(data=decoder_x, act_type=\"sigmoid\",name='activation_x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Joint Loss Function for the Encoder and the Decoder\n",
    "\n",
    "The variational lower bound also called evidence lower bound (ELBO) can be estimated as:\n",
    "\n",
    "\\begin{align}\n",
    "\\mathcal{L}(\\theta,\\phi;x_{(i)}) \\approx \\frac{1}{2}\\left(1+\\log ((\\sigma_j^{(i)})^2)-(\\mu_j^{(i)})^2-(\\sigma_j^{(i)})^2\\right) + \\log p_\\theta(x^{(i)}|z^{(i)})\n",
    "\\end{align}\n",
    "\n",
    "where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\\mathcal{L}$(minus ELBO) for MXNet to minimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define the objective loss function that needs to be minimized\n",
    "KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)\n",
    "loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) \n",
    "                      + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL\n",
    "output = mx.symbol.MakeLoss(sum(loss),name='loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training the model\n",
    "\n",
    "Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. \n",
    "\n",
    "To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set up the log\n",
    "nd_iter.reset()\n",
    "logging.getLogger().setLevel(logging.DEBUG)  \n",
    "\n",
    "# define function to trave back training loss\n",
    "def log_to_list(period, lst):\n",
    "    def _callback(param):\n",
    "        \"\"\"The checkpoint function.\"\"\"\n",
    "        if param.nbatch % period == 0:\n",
    "            name, value = param.eval_metric.get()\n",
    "            lst.append(value)\n",
    "    return _callback\n",
    "\n",
    "# define the model\n",
    "model = mx.mod.Module(\n",
    "    symbol = output ,\n",
    "    data_names=['data'],\n",
    "    label_names = ['loss_label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Train-loss=375.023381\n",
      "INFO:root:Epoch[0] Time cost=6.127\n",
      "INFO:root:Epoch[1] Train-loss=212.780315\n",
      "INFO:root:Epoch[1] Time cost=6.409\n",
      "INFO:root:Epoch[2] Train-loss=208.209400\n",
      "INFO:root:Epoch[2] Time cost=6.619\n",
      "INFO:root:Epoch[3] Train-loss=206.146854\n",
      "INFO:root:Epoch[3] Time cost=6.648\n",
      "INFO:root:Epoch[4] Train-loss=204.530598\n",
      "INFO:root:Epoch[4] Time cost=7.000\n",
      "INFO:root:Epoch[5] Train-loss=202.799992\n",
      "INFO:root:Epoch[5] Time cost=6.778\n",
      "INFO:root:Epoch[6] Train-loss=200.333474\n",
      "INFO:root:Epoch[6] Time cost=7.187\n",
      "INFO:root:Epoch[7] Train-loss=197.506393\n",
      "INFO:root:Epoch[7] Time cost=6.712\n",
      "INFO:root:Epoch[8] Train-loss=195.969775\n",
      "INFO:root:Epoch[8] Time cost=6.896\n",
      "INFO:root:Epoch[9] Train-loss=195.418288\n",
      "INFO:root:Epoch[9] Time cost=6.887\n",
      "INFO:root:Epoch[10] Train-loss=194.739763\n",
      "INFO:root:Epoch[10] Time cost=6.745\n",
      "INFO:root:Epoch[11] Train-loss=194.380536\n",
      "INFO:root:Epoch[11] Time cost=6.706\n",
      "INFO:root:Epoch[12] Train-loss=193.955462\n",
      "INFO:root:Epoch[12] Time cost=6.592\n",
      "INFO:root:Epoch[13] Train-loss=193.493671\n",
      "INFO:root:Epoch[13] Time cost=6.775\n",
      "INFO:root:Epoch[14] Train-loss=192.958739\n",
      "INFO:root:Epoch[14] Time cost=6.600\n",
      "INFO:root:Epoch[15] Train-loss=191.928542\n",
      "INFO:root:Epoch[15] Time cost=6.586\n",
      "INFO:root:Epoch[16] Train-loss=189.797939\n",
      "INFO:root:Epoch[16] Time cost=6.700\n",
      "INFO:root:Epoch[17] Train-loss=186.672446\n",
      "INFO:root:Epoch[17] Time cost=6.869\n",
      "INFO:root:Epoch[18] Train-loss=184.616599\n",
      "INFO:root:Epoch[18] Time cost=7.144\n",
      "INFO:root:Epoch[19] Train-loss=183.305978\n",
      "INFO:root:Epoch[19] Time cost=6.997\n",
      "INFO:root:Epoch[20] Train-loss=181.944634\n",
      "INFO:root:Epoch[20] Time cost=6.481\n",
      "INFO:root:Epoch[21] Train-loss=181.005329\n",
      "INFO:root:Epoch[21] Time cost=6.754\n",
      "INFO:root:Epoch[22] Train-loss=178.363118\n",
      "INFO:root:Epoch[22] Time cost=7.000\n",
      "INFO:root:Epoch[23] Train-loss=176.363421\n",
      "INFO:root:Epoch[23] Time cost=6.923\n",
      "INFO:root:Epoch[24] Train-loss=174.573954\n",
      "INFO:root:Epoch[24] Time cost=6.510\n",
      "INFO:root:Epoch[25] Train-loss=173.245940\n",
      "INFO:root:Epoch[25] Time cost=6.926\n",
      "INFO:root:Epoch[26] Train-loss=172.082522\n",
      "INFO:root:Epoch[26] Time cost=6.733\n",
      "INFO:root:Epoch[27] Train-loss=171.123084\n",
      "INFO:root:Epoch[27] Time cost=6.616\n",
      "INFO:root:Epoch[28] Train-loss=170.239300\n",
      "INFO:root:Epoch[28] Time cost=7.004\n",
      "INFO:root:Epoch[29] Train-loss=169.538416\n",
      "INFO:root:Epoch[29] Time cost=6.341\n",
      "INFO:root:Epoch[30] Train-loss=168.952901\n",
      "INFO:root:Epoch[30] Time cost=6.736\n",
      "INFO:root:Epoch[31] Train-loss=168.169076\n",
      "INFO:root:Epoch[31] Time cost=6.616\n",
      "INFO:root:Epoch[32] Train-loss=167.208973\n",
      "INFO:root:Epoch[32] Time cost=6.446\n",
      "INFO:root:Epoch[33] Train-loss=165.732213\n",
      "INFO:root:Epoch[33] Time cost=6.405\n",
      "INFO:root:Epoch[34] Train-loss=163.606801\n",
      "INFO:root:Epoch[34] Time cost=6.139\n",
      "INFO:root:Epoch[35] Train-loss=161.985880\n",
      "INFO:root:Epoch[35] Time cost=6.678\n",
      "INFO:root:Epoch[36] Train-loss=160.763072\n",
      "INFO:root:Epoch[36] Time cost=8.749\n",
      "INFO:root:Epoch[37] Train-loss=160.025193\n",
      "INFO:root:Epoch[37] Time cost=6.519\n",
      "INFO:root:Epoch[38] Train-loss=159.319723\n",
      "INFO:root:Epoch[38] Time cost=7.584\n",
      "INFO:root:Epoch[39] Train-loss=158.670701\n",
      "INFO:root:Epoch[39] Time cost=6.874\n",
      "INFO:root:Epoch[40] Train-loss=158.225733\n",
      "INFO:root:Epoch[40] Time cost=6.402\n",
      "INFO:root:Epoch[41] Train-loss=157.741337\n",
      "INFO:root:Epoch[41] Time cost=8.617\n",
      "INFO:root:Epoch[42] Train-loss=157.301411\n",
      "INFO:root:Epoch[42] Time cost=6.515\n",
      "INFO:root:Epoch[43] Train-loss=156.765170\n",
      "INFO:root:Epoch[43] Time cost=6.447\n",
      "INFO:root:Epoch[44] Train-loss=156.389668\n",
      "INFO:root:Epoch[44] Time cost=6.130\n",
      "INFO:root:Epoch[45] Train-loss=155.815434\n",
      "INFO:root:Epoch[45] Time cost=6.155\n",
      "INFO:root:Epoch[46] Train-loss=155.432254\n",
      "INFO:root:Epoch[46] Time cost=6.158\n",
      "INFO:root:Epoch[47] Train-loss=155.114027\n",
      "INFO:root:Epoch[47] Time cost=6.749\n",
      "INFO:root:Epoch[48] Train-loss=154.612441\n",
      "INFO:root:Epoch[48] Time cost=6.255\n",
      "INFO:root:Epoch[49] Train-loss=154.137659\n",
      "INFO:root:Epoch[49] Time cost=7.813\n",
      "INFO:root:Epoch[50] Train-loss=153.634072\n",
      "INFO:root:Epoch[50] Time cost=7.408\n",
      "INFO:root:Epoch[51] Train-loss=153.417397\n",
      "INFO:root:Epoch[51] Time cost=7.747\n",
      "INFO:root:Epoch[52] Train-loss=152.851887\n",
      "INFO:root:Epoch[52] Time cost=8.587\n",
      "INFO:root:Epoch[53] Train-loss=152.575068\n",
      "INFO:root:Epoch[53] Time cost=7.554\n",
      "INFO:root:Epoch[54] Train-loss=152.084419\n",
      "INFO:root:Epoch[54] Time cost=6.628\n",
      "INFO:root:Epoch[55] Train-loss=151.724836\n",
      "INFO:root:Epoch[55] Time cost=6.535\n",
      "INFO:root:Epoch[56] Train-loss=151.302525\n",
      "INFO:root:Epoch[56] Time cost=7.148\n",
      "INFO:root:Epoch[57] Train-loss=150.960916\n",
      "INFO:root:Epoch[57] Time cost=7.195\n",
      "INFO:root:Epoch[58] Train-loss=150.603895\n",
      "INFO:root:Epoch[58] Time cost=6.649\n",
      "INFO:root:Epoch[59] Train-loss=150.237795\n",
      "INFO:root:Epoch[59] Time cost=6.222\n",
      "INFO:root:Epoch[60] Train-loss=149.936080\n",
      "INFO:root:Epoch[60] Time cost=8.450\n",
      "INFO:root:Epoch[61] Train-loss=149.514617\n",
      "INFO:root:Epoch[61] Time cost=6.113\n",
      "INFO:root:Epoch[62] Train-loss=149.229345\n",
      "INFO:root:Epoch[62] Time cost=6.088\n",
      "INFO:root:Epoch[63] Train-loss=148.893769\n",
      "INFO:root:Epoch[63] Time cost=6.558\n",
      "INFO:root:Epoch[64] Train-loss=148.526837\n",
      "INFO:root:Epoch[64] Time cost=7.590\n",
      "INFO:root:Epoch[65] Train-loss=148.249951\n",
      "INFO:root:Epoch[65] Time cost=6.180\n",
      "INFO:root:Epoch[66] Train-loss=147.940414\n",
      "INFO:root:Epoch[66] Time cost=6.242\n",
      "INFO:root:Epoch[67] Train-loss=147.621304\n",
      "INFO:root:Epoch[67] Time cost=8.501\n",
      "INFO:root:Epoch[68] Train-loss=147.294314\n",
      "INFO:root:Epoch[68] Time cost=7.645\n",
      "INFO:root:Epoch[69] Train-loss=147.074479\n",
      "INFO:root:Epoch[69] Time cost=7.092\n",
      "INFO:root:Epoch[70] Train-loss=146.796387\n",
      "INFO:root:Epoch[70] Time cost=6.914\n",
      "INFO:root:Epoch[71] Train-loss=146.508842\n",
      "INFO:root:Epoch[71] Time cost=6.606\n",
      "INFO:root:Epoch[72] Train-loss=146.230444\n",
      "INFO:root:Epoch[72] Time cost=7.755\n",
      "INFO:root:Epoch[73] Train-loss=145.970296\n",
      "INFO:root:Epoch[73] Time cost=6.409\n",
      "INFO:root:Epoch[74] Train-loss=145.711610\n",
      "INFO:root:Epoch[74] Time cost=6.334\n",
      "INFO:root:Epoch[75] Train-loss=145.460053\n",
      "INFO:root:Epoch[75] Time cost=7.269\n",
      "INFO:root:Epoch[76] Train-loss=145.156451\n",
      "INFO:root:Epoch[76] Time cost=6.744\n",
      "INFO:root:Epoch[77] Train-loss=144.957674\n",
      "INFO:root:Epoch[77] Time cost=7.100\n",
      "INFO:root:Epoch[78] Train-loss=144.729749\n",
      "INFO:root:Epoch[78] Time cost=6.242\n",
      "INFO:root:Epoch[79] Train-loss=144.481728\n",
      "INFO:root:Epoch[79] Time cost=6.865\n",
      "INFO:root:Epoch[80] Train-loss=144.236061\n",
      "INFO:root:Epoch[80] Time cost=6.632\n",
      "INFO:root:Epoch[81] Train-loss=144.030473\n",
      "INFO:root:Epoch[81] Time cost=6.764\n",
      "INFO:root:Epoch[82] Train-loss=143.776374\n",
      "INFO:root:Epoch[82] Time cost=6.564\n",
      "INFO:root:Epoch[83] Train-loss=143.538847\n",
      "INFO:root:Epoch[83] Time cost=6.181\n",
      "INFO:root:Epoch[84] Train-loss=143.326444\n",
      "INFO:root:Epoch[84] Time cost=6.220\n",
      "INFO:root:Epoch[85] Train-loss=143.078987\n",
      "INFO:root:Epoch[85] Time cost=6.823\n",
      "INFO:root:Epoch[86] Train-loss=142.877117\n",
      "INFO:root:Epoch[86] Time cost=7.755\n",
      "INFO:root:Epoch[87] Train-loss=142.667316\n",
      "INFO:root:Epoch[87] Time cost=6.068\n",
      "INFO:root:Epoch[88] Train-loss=142.461755\n",
      "INFO:root:Epoch[88] Time cost=6.111\n",
      "INFO:root:Epoch[89] Train-loss=142.270438\n",
      "INFO:root:Epoch[89] Time cost=6.221\n",
      "INFO:root:Epoch[90] Train-loss=142.047086\n",
      "INFO:root:Epoch[90] Time cost=8.061\n",
      "INFO:root:Epoch[91] Train-loss=141.855774\n",
      "INFO:root:Epoch[91] Time cost=6.433\n",
      "INFO:root:Epoch[92] Train-loss=141.688955\n",
      "INFO:root:Epoch[92] Time cost=7.153\n",
      "INFO:root:Epoch[93] Train-loss=141.442910\n",
      "INFO:root:Epoch[93] Time cost=7.113\n",
      "INFO:root:Epoch[94] Train-loss=141.279274\n",
      "INFO:root:Epoch[94] Time cost=7.152\n",
      "INFO:root:Epoch[95] Train-loss=141.086522\n",
      "INFO:root:Epoch[95] Time cost=6.472\n",
      "INFO:root:Epoch[96] Train-loss=140.901925\n",
      "INFO:root:Epoch[96] Time cost=6.767\n",
      "INFO:root:Epoch[97] Train-loss=140.722496\n",
      "INFO:root:Epoch[97] Time cost=7.044\n",
      "INFO:root:Epoch[98] Train-loss=140.579295\n",
      "INFO:root:Epoch[98] Time cost=7.040\n",
      "INFO:root:Epoch[99] Train-loss=140.386067\n",
      "INFO:root:Epoch[99] Time cost=6.669\n"
     ]
    }
   ],
   "source": [
    "# training the model, save training loss as a list.\n",
    "training_loss=list()\n",
    "\n",
    "# initilize the parameters for training using Normal.\n",
    "init = mx.init.Normal(0.01)\n",
    "model.fit(nd_iter,  # train data\n",
    "          initializer=init,\n",
    "          # if eval_data is supplied, test loss will also be reported\n",
    "          # eval_data = nd_iter_test,\n",
    "          optimizer='sgd',  # use SGD to train\n",
    "          optimizer_params={'learning_rate':1e-3,'wd':1e-2},  \n",
    "          # save parameters for each epoch if model_prefix is supplied\n",
    "          epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),\n",
    "          batch_end_callback = log_to_list(N/batch_size,training_loss), \n",
    "          num_epoch=100,\n",
    "          eval_metric = 'Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcZGV97/HPt6v3bfZ9BmaEEdlRRwSjUSO5EGIC90oM\nMYkYt/iSxJhrouISlStJNIkmvhKNxusVIooaF9C4gUaMMYADso6AA8MwM8zSs/Z0Ty+1/O4f53RT\n9PSpGmqmunq6v+/Xq15z6jmnqp7ndM3zq2c5z1FEYGZmVoumRmfAzMyOXw4iZmZWMwcRMzOrmYOI\nmZnVzEHEzMxq5iBiZmY1cxCxY0rSP0t677E+diaQ1CHpG5IOSPpyAz7/XZI+fbTHSnqJpK3HNneZ\n+XhM0gVT8VlWm+ZGZ8CmD0mPAa+PiFtqfY+IeFM9jp0hLgOWAAsiojDVHx4Rf1mPY58OSauBTUBL\nI86BHXtuidgRkzSrfnTUobwnAg/XUnnOtnNvxw8HEQNA0r8CJwDfkDQg6e2SVksKSa+T9Djwg/TY\nL0vakXbL/EjS6WXv81lJH0y3XyJpq6S3SdolabukP6jx2AVpV1C/pJ9K+qCkH1cozwsl/UTSfklb\nJL0mTf+hpNeXHfea8vdJy3ulpF8Av5D0CUl/O+G9b5T0v9Pt5ZK+IqlP0iZJb8nIzweAvwB+Oz2/\nr5PUJOk9kjanZb5O0pz0+EnP/YT3HDtnby87Z5dKuljSw5L2SnpX2fHvl/S5Ce9/haTHJe2W9O7J\njq1wjt+Vvu4xSb9blv7rkn6W/q22SHp/2ct+lP67Pz0P56eveYOkn0s6KGmDpOeUveYcSfem37cv\nSmov+6yXS7o7/Tv/RNJZZfveIWlb+p4PSXpZpfJYjSLCDz+ICIDHgAvKnq8GArgO6AI60vTXAj1A\nG/D3wN1lr/ks8MF0+yVAAbgaaAEuBg4B82o49ob00QmcBmwBfpxRjhOBg8DvpO+1ADgn3fdDki67\nsWNfU/4+aXlvBuYDHcAvp5+ldP88YAhYTvIj7E6S4NAKPAN4FLgwI1/vBz5X9vy1wMb0dd3AV4F/\nrXTuJ7zf2Dn7i7ScbwD6gM+nf5/T07yumfj5Ze//L2k5zwZGgFMny2vG534k/Q68GBgETinbf2Z6\nfs4CdgKXTvjc5rL3+y1gG/A8QMDJwIll38k70vM9H/g58KZ037OBXcDzgRxwRXp8G3BK+ndbXva5\nJzX6/9hMfLglYkfi/RExGBFDABHxmYg4GBEjJJXN2WO/oCeRB66OiHxEfAsYIPkPfsTHSsoBrwDe\nFxGHImIDcG2F/L4KuCUivpC+156IuPtplPevImJvWt7/JKn0XpTuuwz474h4gqTSWxQRV0fEaEQ8\nSlIpX36En/O7wEci4tGIGACuAi6f0HX1lHM/iTxwTUTkSYLsQuAf0r/PA8AGkgCR5QMRMRQR9wD3\nVDl2ovdGxEhE3Ar8O/BKgIj4YUTcFxGliLgX+AJJoMnyeuDDEfHTSGyMiM1l+z8WEU9ExF7gG8A5\nafobgU9GxO0RUYyIa0kC4XlAkSSYnCapJSIei4hHnkbZ7Ag5iNiR2DK2ISkn6a8lPSKpn+SXHySV\n12T2xFPHAA6R/Op+OscuIpkEsqVsX/n2RKuAo6kwxt87IoKkcv6dNOlVwPXp9onA8rQrZb+k/cC7\nSAbPj8RyoLyy3ExSzvLXVyonJOesmG6PBZqdZfuHyD7fADvKtiv9bSbaFxGDZc83k5QHSc+X9B9p\nF98B4E1kfz+g+t8rK48nAm+bcP5XkbQ+NgJvJfmRs0vSDZKWH2HZ7GlwELFyWUs6l6e/CrgEuACY\nQ9JNAEk3RL30kXSfrCxLW1Xh+C3ASRn7Bkm6xMYsneSYiefhC8Blkk4k6Tr5StnnbIqIuWWPnoi4\nuELeyj1BUhGOOYGknOVBYLousz1PUlfZ8xNIygNJd9pNwKqImAP8M09+PyYrT6W/VyVbSFph5ee/\nMyK+ABARn4+IF5Kc4wA+VMNnWBUOIlZuJ0n/fCU9JF0Ge0gq47pMBS2X/tL+KvB+SZ2SngW8usJL\nrgcukPRKSc3poPxYF8jdwP9K3+dk4HVH8Pk/A3YDnwa+GxH70113AAfTAdyOtJV2hqTnHWHRvgD8\nqaQ1krpJzuUX4/iZ+voBSa2SXgS8HBi79qUH2BsRw5LOJfnhMaYPKPHU79mngT+T9FwlTk4DdjX/\nArwpbflIUlc6qN8j6RRJvyKpDRgmaZGVjrK8NgkHESv3V8B70q6BP8s45jqSrottJP3tt01R3v6I\npOWzA/hXkgp4ZLIDI+JxkoH5twF7SQLHWF//R4FRkoB5LU92TVXzeZLW1+fLPqdIUnmeQ3Ltw1ig\nyRofmugzaVl+lL5+GPjjI3xto+0A9pG0Pq4nGex+MN33ZuBqSQdJBv2/NPaiiDgEXAP8V/o9Oy8i\nvpymfZ5kQsTXSQbRK4qI9SSTCf4xzctGkokSkIyH/DXJ32QHsJhkzMmOsbEZJ2bHFUkfApZGxBWN\nzovZbOaWiB0XJD1L0llpt8W5JN1QX2t0vsxmO18Fa8eLHpIurOUkXVF/B9zY0ByZmbuzzMysdu7O\nMjOzms347qyFCxfG6tWrG50NM7Pjyp133rk7IhZVO27GB5HVq1ezfv36RmfDzOy4Imlz9aPcnWVm\nZkfBQcTMzGrmIGJmZjVzEDEzs5o5iJiZWc0cRMzMrGYOImZmVrMZf52Imdl0FxEM5YsMjBTIF4NS\nKShNWJKqFFAsxfijFEEEyb/pewznS/QNjNB3cIQ9AyP8+YWnINXzfnEOImZmNRkaLbJp9yAb+wZ4\nYv8QAnJNoiXXRE97M3M7W+hsbWbf4Cg7+4fZdXCEwZECh0aLHMoX2TswOl7hHxzOUzrGyxi25pp4\n80tPprutvtW8g4iZHbcigpFCiUOjxfEKenC0wKGRIvliiXyxRKEU5IslRgslRoslhkaTX/wDwwX6\nh/PsP5TnwFCeUgQdrc10tuTIF0vs6B9mZ/8w/UMFJGhKf9EXI2kpFJ5mrZ9rEl2tOTpbm+lszTG/\nq5W1i7t5wUkLmNPRQldbM12tOVqbm5BETmKsEREBTU2Qa2qiuUk0pflpSo+RQIi25iYW9bSxqKeN\nOR0tdW+FgIOImTVARDA4WmTPwAgDIwVyTUmlOVossat/hO0Hkgp8d/pL/cBQnuacaM01kWsSuwdG\n2dU/TN/ACPlibT/hO1py9HY0M7ejlTkdLTQ3NXFgKM+OA0PkmppY2tvGWSvnMKejlSDpOooImtK8\ndrbmWLOwm5MWd7FyXiciCTCjhRIHhwscGMozOFJgbmcLS3rbmd/ZSlNT/Sv1qeYgYmZV5Ysltuw9\nxKHRIqVI+uQPjRY5MJSnfyhPrkn0tLfQ097MwEiB7fuH2H5geDwY7CrrsilFMDRaZKRQ/Zbn87ta\nWdjdytyOVobzJfqHChRKwcLuVk5atJBFPW30tDeP/8Lvamumsy1HZ0uOtpYczU0i1yRam5tozTXR\n2txER2uOrtZmcnWs0Bd2t9XtvacbBxGzWSZfTLp/BkYK7DgwzI4Dw/QdHGak8GSXz6HRIodGC/QP\nFXikb4BH+wYZLVav9Mu1NjextLedpb3tnL68l96OFnJKumLaWnIs6GplQXcb3W258UHj5iaxuLed\npXPaWdzTRkvOE0inOwcRs+NERFAKGBgpsHdwlL2DoxwYGqV/KOnb3z0wyo4DSQtg98Ao/Wkr4VC+\niEj60IOo2v0jQVfab9/d1syJCzp58SmLeObiHrrbm5NA0JQc09vRQm9HC6VScHC4wMHhPJ2tzSyb\n286CrtYp6ZO3xnIQMWuAQ6MFdvaPsKt/mEP5IqOFEiOFEvsGR9k9MMLugWRc4In9Q2zfP8zgaKHq\n7B0JFve0sbS3nRVz2zl1WQ9zOlrobM0l/fnpcZ0tOTrSALFkTjvL5rSzqLuNjtYcLblk4NaVvx0p\nBxGzGo0WSmw/METfwRGG8kWGRosM5Yv0DxfoH8ozMFJI+/6LDI4U6Ts4wq6Dw+zqH+HgSCHzfZsE\n87vaWDqnjdULunjBSQvpbmseH9Dtaktm9szvamVuZzIo3NvenAwOu/vHppiDiM1qhWKJx/ceon+4\nMH4B19iA8dig8Vg3zYGhZDro/qHRNCCMEBVaBy050d6So70lR2drjkXdbZyytIcXrV3Ekt52lvS2\nsbinnc62HG3NTbQ1NzG3s5V5na11HfQ1O5YcRGzGyhdLbN5ziI27DvKLnQM80jfAUL4IJPPut+4b\nYmPfAKNVZgm15ERve9L3P7ezhUXdbTxraS8r5nawYl4HS3rb6WpNgkVHa46e9mZ621tob8lNRTHN\nGspBxGaEgZEC927dzz1bDnDftv08vHOAzXsGnzKIvGJuBz3tyVc+ApbOaeeFaxeydnE3C3va0ou3\nkusH5nS0JN1EHS20pRd/mdnhHETsuFQqBRu29/PDh3bxw4f6uOvxfeMDzyfM7+SUpT386mlLOHlR\nN2uXdHPy4m46W/11NzvW/L/Kjhtb9x3iBw/u4icb93D7pj3sO5QH4MwVc3jzS05m3ep5nL1yLvO6\nWhucU7PZw0HEpp3hfJHNew6ND25v3DXAd+7fzj1bDwBJt9TLTl3C+c9YwIueuZDFPe0NzrHZ7OUg\nYg1RLAV3b9nPY7sHKaaL2W0/MMTtm/Zy95b9hw12n7VyDu+46FlcdMZS1izsalCuzWwiBxGrq4jg\nwFCeXQdHxpfYWL95Lz94cBe7B0afcmyuSZyxvJcrzj+RM1fOZX5nK70dzel0WLc2zKYjBxE7pvYf\nGuXf79vON+/ZzqO7B9gzMHrYktk97c289JTFXHDaEs5aMYeW5uQq6Z72Zg9+mx1n/D/Wjtqh0QI3\nb9jJN+7Zzq0P7yJfDE5a1MWLn7mIhd1tLOhO7m8wthjfsrntXljPbIZwELGnpVgKHukb4OGdB9m4\na4Cfb+/nRw/vZihfZElvG1ecv5pLn72C05f3+toKs1nAQcQqigju3XqA7z+4i7s27+PuLfsZSNd9\nkpJrMl7x3BX8xlnLed7q+TPypjtmls1BxIAkWPxi1wC3b9rL4EiB4XyRXQdH+MHPd7Gjf5gmwanL\nern02cs5Z9U8Tl3Ww0mLur20h9ks5yAyi/UdHOG+bfu5/dG9fG/DTjbtHnzK/u62Zl5w0gL+/PRT\neNmpi5nb6Yv4zOypHERmmYPDeT5566N85a6tbD8wDEBzkzj/pAW87oVrePEzF7Ggu5X25py7psys\nKgeRWWK0UOKGnz7OP9zyC/YMjnLBqUt43QvXcNbKuZy+vJeuNn8VzOzpc80xQ5VKwc6Dw6x/bB83\nb9jJfzy4i4MjBZ6/Zj6fufhUzl41t9FZNLMZwEHkOBAR4/fV3j3w5O1TB0cKjOST26oeHM6z91Ce\nfYOjbD8wxJZ9Q+NLhyzoauXXzlzKb5y9nBeevNBTb83smHEQaaBiKXhszyAbnujnkb4BettbWDan\nnYU9bTy88yA/3bSXux7fz44Dw4wWs2+c1KRkEHxBdxvzOltYu7iHC05dwsr5nZy6tIdnnzDPd8oz\ns7pwEJliEcFPH9vHl9dv4dv37xi/5mIyi3raWHfiPC4+cxnzu1qY19nKwu625NHTSk97csOk5ia5\ndWFmDeEgMkW27R/ia3dt5St3bWPT7kG625q5+MylnLtmAacu6+Hkxd0MjhR5Yv8QfQMjPGNhFyfM\n73RwMLNpzUGkjvYOjnLzhh18457t/Ncju4mAc1fP58qXnszFZy49bLHBtuYc831DJTM7jjQkiEj6\nG+A3gFHgEeAPImJ/uu8q4HVAEXhLRHw3TX8u8FmgA/gW8CcREYe/e2Pt6h/mext28p37d/Dfj+6h\nWApWze/gLb+yllc8ZyUnLOhsdBbNzI6ZRrVEbgauioiCpA8BVwHvkHQacDlwOrAcuEXSMyOiCHwC\neANwO0kQuQj4dkNyP8Hjew7x3Qd28N0HdnDn4/uIgDULu3jTi5/Br52xzIsRmtmM1ZAgEhHfK3t6\nG3BZun0JcENEjACbJG0EzpX0GNAbEbcBSLoOuJQGBpEtew/x9Z9t49/v286DOw4CcNqyXv70gmdy\n0RlLWbu424HDzGa86TAm8lrgi+n2CpKgMmZrmpZPtyemT0rSG4E3ApxwwgnHMq/8ZONuPnLzw6zf\nvA+A562ex3t+/VQuPH0pq+a7q8rMZpe6BRFJtwBLJ9n17oi4MT3m3UABuP5YfnZEfAr4FMC6deuO\n2bjJjgPD/OHn7mRORwt/fuEpXHLOclbOc+Aws9mrbkEkIi6otF/Sa4CXAy8rGyDfBqwqO2xlmrYt\n3Z6YPmUignd+9V7yxRKfe93zWb2wayo/3sxsWmrIPUolXQS8HfjNiDhUtusm4HJJbZLWAGuBOyJi\nO9Av6TwlAw2vBm6cyjz/251b+eFDfbzjomc5gJiZpRo1JvKPQBtwczr4fFtEvCkiHpD0JWADSTfX\nlenMLIA38+QU328zhYPqOw4Mc/U3N3Du6vlccf7qqfpYM7Npr1Gzs06usO8a4JpJ0tcDZ9QzX1k+\n9J0HyRdLfPiys3yPDTOzMg3pzjrebN4zyLoT57sby8xsAgeRI1AoBc05t0DMzCZyEDkC+WLQ3ORT\nZWY2kWvGI1AolmhxS8TM7DAOIkcg6c7yqTIzm8g14xHIF0u0eFaWmdlhHESOQKHogXUzs8k4iByB\nQqnk7iwzs0m4ZjwC+WK4O8vMbBIOIkegUHRLxMxsMq4Zj0DeFxuamU3KQeQIFIolWnyxoZnZYVwz\nVlEqBaXALREzs0k4iFSRL5UAaPGYiJnZYVwzVlEoJjddbPbsLDOzwziIVDEeRNwSMTM7jGvGKsa6\ns9wSMTM7nINIFU+2RBxEzMwmchCpIl9MB9Y9xdfM7DCuGasoltwSMTPL4iBSRWFsTMQD62Zmh3HN\nWEU+HRPxAoxmZodzEKnCU3zNzLK5ZqxifIqvx0TMzA7jIFJFYbw7y6fKzGwi14xVFIpuiZiZZXEQ\nqSKfTvFtcRAxMzuMg0gV4y0Rd2eZmR3GNWMVeS97YmaWyUGkioLvJ2Jmlsk1YxW+n4iZWTYHkSrG\nF2B0S8TM7DCuGasoeAFGM7NMDiJVeHaWmVk214xVjC/A6JaImdlhHESq8FLwZmbZXDNWkffsLDOz\nTA4iVXiKr5lZNgeRKsa6s3IOImZmh2lIEJH0fyTdK+luSd+TtLxs31WSNkp6SNKFZenPlXRfuu9j\nkqakVi+UgpacmKKPMzM7rjSqJfI3EXFWRJwDfBP4CwBJpwGXA6cDFwEfl5RLX/MJ4A3A2vRx0VRk\ntFAseXqvmVmGhtSOEdFf9rQLiHT7EuCGiBiJiE3ARuBcScuA3oi4LSICuA64dCrymi+GLzQ0M8vQ\n3KgPlnQN8GrgAPDSNHkFcFvZYVvTtHy6PTE9673fCLwR4IQTTjiqfBZKJS95YmaWoW61o6RbJN0/\nyeMSgIh4d0SsAq4H/uhYfnZEfCoi1kXEukWLFh3VexWK4ZlZZmYZ6tYSiYgLjvDQ64FvAe8DtgGr\nyvatTNO2pdsT0+suXwy3RMzMMjRqdtbasqeXAA+m2zcBl0tqk7SGZAD9jojYDvRLOi+dlfVq4Map\nyGuhVPKYiJlZhkaNify1pFOAErAZeBNARDwg6UvABqAAXBkRxfQ1bwY+C3QA304fdefuLDOzbA0J\nIhHxigr7rgGumSR9PXBGPfM1mXzRA+tmZllcO1ZRKHmKr5lZFgeRKvK+2NDMLFPV2lHSGZKuk7Q+\nfVwr6aypyNx0UCiG7yViZpahYhBJr+n4GvBD4LXp41bgK2PXe8x0hZJbImZmWaoNrF8N/GpEPFaW\ndq+kH5BMsZ2SabaNlC8G7S1uiZiZTabaT+zmCQEEgDStpR4Zmm687ImZWbZqtWNB0mGLT0k6keQ6\njhnP14mYmWWr1p31PuAWSX8J3JmmrQPeCbyjnhmbLvJFX7FuZpalYhCJiK9L2gS8DfjjNHkD8MqI\nuKfemZsOCqXwwLqZWYaqV6ynweLVU5CXaang+4mYmWWqNsV3oaT3SXqLpG5Jn0iXc79R0slTlclG\nKpRKtLglYmY2qWq14+eBNtLVdIFNwGUkt7T9dH2zNj24JWJmlq1ad9aSiHhXuvz65oj4cJr+oKQr\n65y3acELMJqZZatWOxYB0vua756wr1SXHE0zycC6WyJmZpOp1hJ5hqSbAJVtkz5fU9ecTRNJd5Zb\nImZmk6kWRMrXx/rbCfsmPp+R8qWSF2A0M8tQ7TqRW7P2SfoiyWKMM1axFETg60TMzDIcTe14/jHL\nxTSVLybDPp6dZWY2Of/ErqBQCgB3Z5mZZajYnSXpOVm7mAWr+BbGWiLuzjIzm1S1gfW/q7DvwWOZ\nkekoX3RLxMyskmoD6y+dqoxMR4XS2JiIWyJmZpOptnbW28u2f2vCvr+sV6ami0LaEvHFhmZmk6v2\nE/vysu2rJuy76BjnZdoZm53lZU/MzCZXrXZUxvZkz2ecsdlZnuJrZja5akEkMrYnez7j5D07y8ys\nomqzs86W1E/S6uhIt0mft9c1Z9NAwbOzzMwqqjY7KzdVGZmOxmZn5TywbmY2KffTVPDkdSI+TWZm\nk3HtWEGx5Cm+ZmaVOIhU8OQCjD5NZmaTce1YgQfWzcwqcxCpYHzZE0/xNTOblGvHCrwAo5lZZQ4i\nFXgBRjOzylw7VpD3AoxmZhU5iFRQ8HUiZmYVuXas4MnuLLdEzMwm4yBSwfjAumdnmZlNqqG1o6S3\nSQpJC8vSrpK0UdJDki4sS3+upPvSfR+TVPfmwfg91t0SMTObVMOCiKRVwP8AHi9LO43kRlink9z0\n6uOSxhaB/ATwBmBt+qj7TbF8PxEzs8oa2RL5KPB2nnpfkkuAGyJiJCI2ARuBcyUtA3oj4raICOA6\n4NJ6Z3D8zobuzjIzm1RDakdJlwDbIuKeCbtWAFvKnm9N01ak2xPTs97/jZLWS1rf19dXcz4LxaBJ\n0OQpvmZmk6p2U6qaSboFWDrJrncD7yLpyqqLiPgU8CmAdevW1XwHxnyp5AsNzcwqqFsQiYgLJkuX\ndCawBrgnHRtfCdwl6VxgG7Cq7PCVadq2dHtiel0VikGLWyFmZpmm/Gd2RNwXEYsjYnVErCbpmnpO\nROwAbgIul9QmaQ3JAPodEbEd6Jd0Xjor69XAjfXOa6HoloiZWSV1a4nUIiIekPQlYANQAK6MiGK6\n+83AZ4EO4Nvpo67ypfCSJ2ZmFTQ8iKStkfLn1wDXTHLceuCMKcoWMNYScRAxM8vivpoKCqXwvUTM\nzCpwDVlBoRi+l4iZWQUOIhUUPMXXzKwi15AV5IseWDczq8RBpIJCseR7iZiZVeAasoJCKTw7y8ys\nAgeRCvLFkhdfNDOrwDVkBYWiWyJmZpU4iFSQL4VnZ5mZVeAasoJCseQFGM3MKnAQqcDdWWZmlTmI\nVOD7iZiZVeYasgLfT8TMrDIHkQp8PxEzs8pcQ1aQL3kBRjOzShxEKigUS14K3sysAteQFXh2lplZ\nZQ4iFeRLXoDRzKwS15AVFIpBzrOzzMwyOYhkiAgKJU/xNTOrxEEkQ7EUAJ7ia2ZWgWvIDIXxIOKW\niJlZFgeRDPliCcD3EzEzq8A1ZIZC0S0RM7NqHEQy5EtJS8RjImZm2VxDZhhriXh2lplZNgeRDE92\nZ/kUmZllcQ2ZYaw7ywswmpllcxDJMN4S8ewsM7NMriEzjE3x9ewsM7NsDiIZxi42dHeWmVk2B5EM\nhbGWiLuzzMwyuYbMkPfFhmZmVTmIZCiMz87yKTIzy+IaMsOTs7PcEjEzy+IgkmF8AUa3RMzMMrmG\nzOCl4M3MqnMQyTB+nYi7s8zMMjmIZPAV62Zm1TWkhpT0fknbJN2dPi4u23eVpI2SHpJ0YVn6cyXd\nl+77mKS6NhGK7s4yM6uqkT+zPxoR56SPbwFIOg24HDgduAj4uKRcevwngDcAa9PHRfXMXN5TfM3M\nqppuNeQlwA0RMRIRm4CNwLmSlgG9EXFbRARwHXBpPTPiKb5mZtU1Moj8saR7JX1G0rw0bQWwpeyY\nrWnainR7YvqkJL1R0npJ6/v6+mrK3JMLME63OGtmNn3UrYaUdIuk+yd5XELSNfUM4BxgO/B3x/Kz\nI+JTEbEuItYtWrSopvfwAoxmZtU11+uNI+KCIzlO0r8A30yfbgNWle1emaZtS7cnpteNF2A0M6uu\nUbOzlpU9/Z/A/en2TcDlktokrSEZQL8jIrYD/ZLOS2dlvRq4sZ55HFuA0S0RM7NsdWuJVPFhSecA\nATwG/CFARDwg6UvABqAAXBkRxfQ1bwY+C3QA304fdVMolcg1iTrPJDYzO641JIhExO9X2HcNcM0k\n6euBM+qZr3KFYnhmlplZFe7wz5Avhq8RMTOrwrVkhkKp5KvVzcyqcBDJkC+GZ2aZmVXhWjJDoVjy\nzCwzsyocRDIUSuHuLDOzKhxEMuSLJVrcnWVmVpFryQyFolsiZmbVOIhkKJRKHlg3M6vCtWSGvFsi\nZmZVOYhkSFoiDiJmZpU4iGRIxkR8eszMKnEtmaFQCl8nYmZWhYNIhkLRA+tmZtW4lsyQLMDoloiZ\nWSUOIhk8xdfMrLpG3ZRq2nvR2kUsm9Pe6GyYmU1rDiIZ3vvy0xqdBTOzac/9NWZmVjMHETMzq5mD\niJmZ1cxBxMzMauYgYmZmNXMQMTOzmjmImJlZzRxEzMysZoqIRuehriT1AZtrfPlCYPcxzM7xYDaW\nGWZnuWdjmWF2lruWMp8YEYuqHTTjg8jRkLQ+ItY1Oh9TaTaWGWZnuWdjmWF2lrueZXZ3lpmZ1cxB\nxMzMauYgUtmnGp2BBpiNZYbZWe7ZWGaYneWuW5k9JmJmZjVzS8TMzGrmIGJmZjVzEJmEpIskPSRp\no6R3Njo/9SJplaT/kLRB0gOS/iRNny/pZkm/SP+d1+i8HmuScpJ+Jumb6fPZUOa5kv5N0oOSfi7p\n/Jlebkl/mn6375f0BUntM7HMkj4jaZek+8vSMssp6aq0fntI0oVH89kOIhNIygH/BPwacBrwO5Jm\n6m0OC8CgrOS/AAAEb0lEQVTbIuI04DzgyrSs7wS+HxFrge+nz2eaPwF+XvZ8NpT5H4DvRMSzgLNJ\nyj9jyy1pBfAWYF1EnAHkgMuZmWX+LHDRhLRJy5n+H78cOD19zcfTeq8mDiKHOxfYGBGPRsQocANw\nSYPzVBcRsT0i7kq3D5JUKitIyntteti1wKWNyWF9SFoJ/Drw6bLkmV7mOcAvA/8XICJGI2I/M7zc\nJLcA75DUDHQCTzADyxwRPwL2TkjOKuclwA0RMRIRm4CNJPVeTRxEDrcC2FL2fGuaNqNJWg08G7gd\nWBIR29NdO4AlDcpWvfw98HagVJY208u8BugD/l/ajfdpSV3M4HJHxDbgb4HHge3AgYj4HjO4zBNk\nlfOY1nEOIoakbuArwFsjor98XyRzwGfMPHBJLwd2RcSdWcfMtDKnmoHnAJ+IiGcDg0zoxplp5U7H\nAC4hCaDLgS5Jv1d+zEwrc5Z6ltNB5HDbgFVlz1emaTOSpBaSAHJ9RHw1Td4paVm6fxmwq1H5q4Nf\nAn5T0mMkXZW/IulzzOwyQ/Jrc2tE3J4+/zeSoDKTy30BsCki+iIiD3wVeAEzu8zlssp5TOs4B5HD\n/RRYK2mNpFaSAaibGpynupAkkj7yn0fER8p23QRckW5fAdw41Xmrl4i4KiJWRsRqkr/tDyLi95jB\nZQaIiB3AFkmnpEkvAzYws8v9OHCepM70u/4yknG/mVzmclnlvAm4XFKbpDXAWuCOWj/EV6xPQtLF\nJP3mOeAzEXFNg7NUF5JeCPwncB9Pjg+8i2Rc5EvACSTL6L8yIiYO2h33JL0E+LOIeLmkBczwMks6\nh2QyQSvwKPAHJD8kZ2y5JX0A+G2SmYg/A14PdDPDyizpC8BLSJZ83wm8D/g6GeWU9G7gtSTn5a0R\n8e2aP9tBxMzMauXuLDMzq5mDiJmZ1cxBxMzMauYgYmZmNXMQMTOzmjmImE1Tkl4ytsqw2XTlIGJm\nZjVzEDE7SpJ+T9Idku6W9Mn0XiUDkj6a3svi+5IWpceeI+k2SfdK+trYPR4knSzpFkn3SLpL0knp\n23eX3QPk+vTKa7Npw0HE7ChIOpXkiuhfiohzgCLwu0AXsD4iTgduJbmCGOA64B0RcRbJSgFj6dcD\n/xQRZ5Os7zS2+uqzgbeS3NvmGSRrf5lNG82NzoDZce5lwHOBn6aNhA6She5KwBfTYz4HfDW9p8fc\niLg1Tb8W+LKkHmBFRHwNICKGAdL3uyMitqbP7wZWAz+uf7HMjoyDiNnREXBtRFz1lETpvROOq3V9\noZGy7SL+P2vTjLuzzI7O94HLJC2G8ftan0jyf+uy9JhXAT+OiAPAPkkvStN/H7g1vavkVkmXpu/R\nJqlzSkthViP/qjE7ChGxQdJ7gO9JagLywJUkN306N923i2TcBJIluf85DRJjK+lCElA+Kenq9D1+\nawqLYVYzr+JrVgeSBiKiu9H5MKs3d2eZmVnN3BIxM7OauSViZmY1cxAxM7OaOYiYmVnNHETMzKxm\nDiJmZlaz/w/UuBmpwDVyRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x145e16898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ELBO = [-training_loss[i] for i in range(len(training_loss))]\n",
    "plt.plot(ELBO)\n",
    "plt.ylabel('ELBO');plt.xlabel('epoch');plt.title(\"training curve for mini batches\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "arg_params = model.get_params()[0]\n",
    "\n",
    "# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch\n",
    "# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)\n",
    "# assert sym.tojson() == output.tojson()\n",
    "\n",
    "e = y.bind(mx.cpu(), {'data': nd_iter_test.data[0][1],\n",
    "                     'encoder_h_weight': arg_params['encoder_h_weight'],\n",
    "                     'encoder_h_bias': arg_params['encoder_h_bias'],\n",
    "                     'mu_weight': arg_params['mu_weight'],\n",
    "                     'mu_bias': arg_params['mu_bias'],\n",
    "                     'logvar_weight':arg_params['logvar_weight'],\n",
    "                     'logvar_bias':arg_params['logvar_bias'],\n",
    "                     'decoder_z_weight':arg_params['decoder_z_weight'],\n",
    "                     'decoder_z_bias':arg_params['decoder_z_bias'],\n",
    "                     'decoder_x_weight':arg_params['decoder_x_weight'],\n",
    "                     'decoder_x_bias':arg_params['decoder_x_bias'],                \n",
    "                     'loss_label':label})\n",
    "\n",
    "x_fit = e.forward()\n",
    "x_construction = x_fit[0].asnumpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADSCAYAAACvmc1VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUXGWZ7/HfkyvkSkKHJARICDAkxMkBaUm4CZrAwQku\nRAdnogzxNjhneRhdOjOiZ80ZdIYj5yxvzNHlDB4QFEcFuSsjIANyRxIMQhJCQi6E0Em6SUI693R4\nzx+1w5Sp56F7d1d1d9X+ftbKSvev3q7au/dT1W9X7+fdllISAAAAUEQD+noDAAAAgL7CZBgAAACF\nxWQYAAAAhcVkGAAAAIXFZBgAAACFxWQYAAAAhcVkuB8ws/9nZl/u6+0AesLMpphZMrNBwe1LzOzc\nXt4soFPULuoRdVs9TIYlmdn2sn9vmtmuss8/WuvHTyl9KqX0v2r9OKgfZrbGzOb29XZUU0ppRkrp\n4b7eDtQWtYt6RN0Wm/vbRNGklEYc+NjM1kj6VErp19F4MxuUUurojW0DaoU6Rr2idlGPqNv+i3eG\nu8DM/snMfmZmPzGzdkmXmtnNZnZV2Zi52UT6wOdHmdkdZtZqZqvN7DNvc/9v3deB+zGzL2Vf+5qZ\nvd/MLjSzFWa22cz+ruxrTzezp8xsq5m1mNk/m9ngstvfZ2YvmdkbZvZ/zexxM/tY2e2fMrMXzWyL\nmf27mR1dpW8baiSrhcXZMX/CzGaW3Xalmb1sZu1mttTMLi677WPZ8f+Wmb0u6aose8zMvp7VwGoz\ne1/Z14w2s+uz2lqfPRcGZrcNzL6uzcxWSZrXyXa/9c6LmV1lZrdmtd9uZs+b2R9ldb/JzNaZ2fll\nX/txM1uWjV1lZp8+6L7/LtvG17KaTmZ2fHbb0Gw7XzGzjWb2L2Z2aM+OArqD2qV26xF12/h1y2S4\n6y6W9G+SRkv62dsNNLMBkn4h6RlJkySdJ+lvzWxOFx/rKJWOzZGS/lHS9ZL+XNIpks6V9FUzOyYb\n2yHps5KaJJ0p6QJJn8624whJt0j62+z21ZJOK9vOD2W3XSRpnKSns31EP2Vmp0i6QaVjfLikf5V0\nt5kNzYa8LOlsler0K5JuNrOJZXcxS9IqSeMlXV2WLVepRv6PpOvNzLLbblSpxo5Xqf7Ol/Sp7La/\nlHRhljdL+tOcu/N+ST+SNEbS7yTdp1LdT5L01WzfDtiUPdYoSR+X9C0ze2f2PblA0uclzc2289yD\nHucaSX8k6eTs9kmS/mfObUUPUbvUbj2ibgtStykl/pX9k7RG0tyDsn+S9B8HZTdLuqrs87mS1mQf\nnylp1UHj/17S94PHfOu+svvZLmlg9vkYSUnSqWXjn5N0YXBffyPp1uzjT0h6tOw2k9Qi6WPZ5w9I\nWlB2+yBJeyRN6uvjUPR/Xh1m+fck/eNB2XJJ5wT3s1jSRdnHH5P0ykG3f0zSyrLPh2X1NkGlF+89\nkg4tu32+pIeyj/9D0l+V3XZ+9rWDOtsnSVdJeqDstvcfVPcjs/s6LLivOyV9Nvv4BklfK7vt+Oxr\nj89qfoek48puP13S6r4+xo36j9qlduvxH3Vb7LrlnOGuW5dj7GRJx5jZ1rJsoKSHu/j1bSml/dnH\nu7L/N5bdvkvSCEkys2mSviHpVJWeVINUeodXKr2z/NZ2p5SSmb160HZ+18yuLcveVOmd6fVd3Fb0\nrsmSFpjZFWXZEJWOtczsMpV+Y5+S3TZCpXcfDvDqeMOBD1JKO7M3KEZIGitpsKSW/3zTQgPK7uMP\n6kvS2pz7cnBNe3U/QtLW7M+I/6DSuw0DVKr158u2Y2HZfZVv07hs7KKyfTCVno/oXdQutVuPqNsC\n1C2T4a5LB32+Q6UDfsCEso/XSVqRUppe860q/VnjKUl/llLabmZ/o9KfNqTSu8Dl5wCZSn+uKN/O\nv08pve1pH+hX1km6OqV09cE3mNlkSd+XNEfSkyml/Wa2WKUXogMOruPOHmuPpKbkN320SCo/x/wY\nZ0yPZX+OvE3SZZLuSintM7M79Z/71aLSL3AHlG9Tm0ov8jNSSvyC17eoXWq3HlG3BahbzhnuvsWS\n5pnZmOz8oL8uu+1JSXvN7Atmdkh20vsfm9mpNdiOkZLekLTDzKYrO1848wtJ77RSA94glc4tHld2\n+79I+h/Z18nMDjOzvOcgoXYGZ/Vz4N8glV54/8rMZlnJcDObZ2YjJQ1X6YW3VSo1QEh6R3cfPKXU\nIul+Sd8ws1FmNsDMjjOzc7Iht0j6ays1i46RdGUP9vXtDJE0VKX96sjesTi/7PZbJH3czKab2TCV\nTkk6sA9vqvQ9+1Z2Dr3MbJKZ/dcabStKqN0Sare+ULclhatbJsPdd6OkZSr9meJXkn564IbsN7o/\nUalZbY1Kvyn9q0onolfbFyQtkNSePcZb7/KmlDZK+jNJ35T0uqTjVDppfk92+63Zbbea2TZJv5fU\nrwu2YO5V6TfsA/+uSiktVKmJ4juStkhaqdI5aEopLVXplJknVfpz2B9LeryH23CZSi+MS7PH+7mk\nA80h31epAeM5Sc9Kur2Hj+VKKbWr9MvmLdk2fETS3WW3/7ukf5b0kErfj6eym/Zk/3/xQJ7V+a8l\nnViLbcVbqF1Ru3WIulUx69ayk5tRAFZanuU1SX+aUnq0r7cHqIXsLx0vSBoa/KkR6JeoXdSjRqhb\n3hlucGZ2QXb6w1CV/pSxT9Jv+3izgKoys4uttLblGEn/W9I99fqijGKhdlGPGq1umQw3vrNUWuOw\nVaVTIC5OKe15+y8B6s6nVVoX82VJ+yX9t77dHKDLqF3Uo4aqW06TAAAAQGHxzjAAAAAKq0eT4ex8\n1OVmttLMarXEBwAAAFAT3T5NIluZ4CVJ50l6VdIzkuZnS424mpqa0pQpU7r1eMABixYtakspjet8\nZPVQu+ipNWvWqK2tzTofWT3ULaqB11zUq67Wbk+uQHeaStfXXiVJZvZTSReptDaea8qUKVq4cGF0\nM9AlZpb3EpQ9Ru2ip5qbm3v9MalbVAOvuahXXa3dnpwmMUl/eD3qV/WHl/oFAAAA+rWaN9CZ2eVm\nttDMFra2ttb64YCqoXZRj6hb1CtqF32lJ5Ph9ZKOLvv8qCz7Ayml61JKzSml5nHjevWUI6BHqF3U\nI+oW9YraRV/pyTnDz0g6wcyOVWkS/OcqXb8aAAAAVdAX14Mw69Ve3z7X7clwSqnDzP67pPskDZR0\nQ0ppSdW2DAAAAKixnrwzrJTSvZLurdK2AAAAAL2KK9ABAACgsJgMAwAAoLCYDAMAAKCwenTOMAAA\nQNFFKz54+ZtvvumO7ejocPPdu3e7+bZt29x88+bNbv7GG2+4+cCBAyuyCRMmuGPHjh3r5sOGDXPz\nwYMHu/mAAf57sX21igXvDAMAAKCwmAwDAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsFhNAgAAoEy0\nOkQkWiFi//79Fdm+ffvcsVu2bHHzVatWufmDDz7o5kuWLMl1/+PHj6/IZs6c6Y6dO3eum0+dOtXN\nq7HKRG+sMME7wwAAACgsJsMAAAAoLCbDAAAAKCwmwwAAACgsGugAAHUnb4NTXn11WVj0b3nrzhsf\nXXY5urzyokWL3Pzpp5928xUrVrh51MzmXXo5uuzykCFD3LyWou95NZ+jvDMMAACAwmIyDAAAgMJi\nMgwAAIDCYjIMAACAwmIyDAAAgMLq0WoSZrZGUruk/ZI6UkrN1dgoAMVV61UCelsRViWoRof92+Xe\nJW2jy99GvMu8vl0eHbe8216Nx8ybo+eq9b3Ncz/t7e1u/tprr7l5W1ubm0erRkybNs3NZ82aVZGd\neuqp7thx48a5ebTKRFTrefRGnVdjabX3pJT8IwIAAAD0Y5wmAQAAgMLq6WQ4Sfq1mS0ys8ursUEA\nAABAb+npZPislNLJkt4n6TNm9u6DB5jZ5Wa20MwWtra29vDhgN5D7aIeUbeoV9Qu+kqPJsMppfXZ\n/5sk3SHpNGfMdSml5pRSc3TiNdAfUbuoR9Qt6hW1i77S7QY6MxsuaUBKqT37+HxJX63algENIk+X\ned5O9aijPk+nfX/rVM/zuNXa9lrua72ujlGNuo3qsKOjw82jbvrNmzdXZNu3b3fH7tu3z82jYxx1\n3kf5oEH+j00vP/TQQ92xUR515Odd2YJVJmon77Hw7Nq1y81feuklN3/llVfcPKrRGTNmuPmcOXPc\nvLm5ciGwsWPHumOHDx/u5tHzIu/KKX2lJ6tJjJd0R7ZDgyT9W0rpV1XZKgAAAKAXdHsynFJaJem/\nVHFbAAAAgF7F0moAAAAoLCbDAAAAKCwmwwAAACisalyOud956qmn3Pzaa69180mTJrl51PG7YMGC\niizqvIxy1K9qrPiwd+9ed+y2bdvcPFpzc+vWrW4eddR79u/f7+YDBw5087wrB+S9f68refDgwe7Y\nqLN5woQJbj5mzBg3j+4/6oRuJHnqOe+x37Nnj5u3tLS4+SOPPOLmzz//fEX2+uuvu2Oj59bQoUPd\nfPz48W4eLe01atQoN/dq68QTT3THHnPMMW5e7x35qOStnLJ27Vp37GOPPebmmzZtcvPode7MM890\n89mzZ7t5U1NTRZb3NTHvyj79rXYb/5UeAAAACDAZBgAAQGExGQYAAEBhMRkGAABAYTEZBgAAQGE1\n5GoS3moPkrRixYqq3P/VV19dkY0ePdodG3Vv1oMpU6a4+Ze+9CU3jzqkG03UUe91DUt+R/2rr77q\njn344Yfd/Omnn3bzDRs2uPmWLVvcfOfOnRVZtBJA1DUcrQ7h3ffb3f+QIUPc3OvWj1YC8LqgJens\ns89280svvdTNo5UDvI7n/tYFXStenUfHMqr9aMWHX/ziF25+//33u/mqVasqssMOO8wdG71uRcc4\nqqFI9HPEq4vo+3X44Ye7+YgRI3JtC3pf3hVV2traKrI777zTHfvcc8+5ebSy1fTp0938tNNOc/Po\nORC9vnry7n80vr/hnWEAAAAUFpNhAAAAFBaTYQAAABQWk2EAAAAUFpNhAAAAFFZDriYRdWouXrzY\nzWfMmOHmS5YscXOvs/+uu+5yx953331ufuyxx7r56tWr3Twv7xr3EydOdMeuW7cu131H3dpf/OIX\nc91PvYq6Y6NVFnbs2FGRrVy50h3729/+1s2ja9ZHq0Z4K1hIfl14mRRfmz7v/kf3f8ghh7i5t0/R\nfkai1Q3mzp3r5lF3/8CBA3M9bn8WHbc83d55V1JpaWlx86VLl7r5+vXr3fzII4+syM455xx3bN5O\n+qhTv7W11c1/+ctfurn38yLq0m9ubnbzeum8L7Jo1YRt27a5+e23316RRfMC72eFJE2bNs3N58yZ\n4+be80WK6zFaOcgTvc5HeXTfUe6tyhI9L6q5sg/vDAMAAKCwmAwDAACgsJgMAwAAoLCYDAMAAKCw\nOp0Mm9kNZrbJzF4oy8aa2QNmtiL7f0xtNxMAAACovq6sJnGjpO9I+mFZdqWkB1NK15jZldnn/WYp\ngeh63VEemTlzppvPnz+/IrvmmmvcsWvWrHHzaDWJVatWdW3jOjFkyJCKLFpNItqWqJs66mxtdJ11\nekedrV7X7OjRo92xRxxxhJtHtbt9+3Y3j+7f6zKeMGGCOzZaYSFaZWLnzp25xkfdxIsWLarI7rnn\nHnfs5s2bc21L3q7kanQrV7Pjub+KOuzzHocTTzzRzWfPnl2RXXDBBe7YqG6jVSPyruAQrZzx4osv\nVmQjRoxwx0arBrCaRP8RHYvdu3e7+RNPPOHmt956a0W2YcMGd2w055g3b56bH3/88W4+bNgwN8+z\ngkP0nI7qf9euXW4erRoUvS56qw/1xmtop+8Mp5QekXTwT5yLJN2UfXyTpA9UebsAAACAmuvuOcPj\nU0oHFpDcIGl8lbYHAAAA6DU9bqBLpb8lhH/bMbPLzWyhmS2M/uwO9EfULuoRdYt6Re2ir3R3MrzR\nzCZKUvb/pmhgSum6lFJzSqk5ugIQ0B9Ru6hH1C3qFbWLvtLdyzHfLWmBpGuy//1rERdIdJJ43maz\nvE1+eXiXkZaktrY2N581a5abn3/++VXbpnpy4CT+PI1ykt/MMHXqVHfsBz7gn35/xhlnuHnU5BE1\nS3rNcsOHD3fHRo1vkaixIhI1/3n5vffe646NtnHSpEluPnbsWDfPcznSelWNJpTo+xRdtjo6Pk1N\nTW4e1e2ZZ55ZkUXNplGjXLSN+/btc/OoPqNLRntNUVu3bnXH5tUbl6Mtquh7G9XF6tWr3fynP/2p\nm3tN9FHtfuhDH3Jzr/4ladSoUW4ePU+jffUupRw1xG3a5L/3Gc0joktARz8DvX3ymuqkfA2BnenK\n0mo/kfSkpBPN7FUz+6RKk+DzzGyFpLnZ5wAAAEBd6fSd4ZRS5TpiJXOqvC0AAABAr2r8vw0CAAAA\nASbDAAAAKCwmwwAAACis7q4mgX7Ou9znxRdf7I6NLrv47W9/282jbu2iiDpVo25179LY0SVjR44c\n6ebRZTcj0THyuvur1ZEb1VHeS/W+8sorFVnU2Rx1U0fd19H3vZpdyY0sb+1H9Tx+vH+dpqjz3Luf\n6DEjXse8JG3bts3Nly5d6ubLly938z179lRk0eol0epD6H3RCgvt7e1u/qtf/crNf/e737m595p7\n7rnnumPnzPFbsaLXuUh0yei9e/e6uff6unbtWnfsM8884+be67YUrygTfQ9OP/30iizaf+9na3fx\nzjAAAAAKi8kwAAAACovJMAAAAAqLyTAAAAAKi8kwAAAACovVJBrUjTfeWJFt2LDBHRt12E+ePLma\nm9Tw8nTaR6sXRNdgz9t9HnXa13J1hOi+Ozo63HzlypVu/vjjj1dkXqe+5HceS9LZZ5/t5tFqBdHx\nKLI8tRJ9/6LVJI444gg3jzr4t2zZUpFFxzLqMI9qKKrDhQsXunlra6ube9+v0aNHu2Or1QUfrYTA\nKiiVou9VtMJCtDrEww8/7ObR6jhTp06tyObNm+eOHTNmjJtHK6Fs3brVzTdv3uzm0fOrpaWlIluy\nZIk79qWXXnLzaPWJaDWh6Pl45JFHVmTTpk1zx7KaBAAAAFAFTIYBAABQWEyGAQAAUFhMhgEAAFBY\nTIYBAABQWKwmUedefvllN//85z/f5ft48skn3XzChAnd2qZGV41O7aj7Pup4jlaHqEY3eXQfeUVd\nw95KAJJ02223ufnq1asrsqgr/4ILLnDzo446ys2j1TqKLKoVry6isXlXk2hqanJzr6tdkp577rmK\nbN26de7Y6BhHdbh8+XI3f+GFF9w86tT3RN+vaHWA6LkYPbei14Wi876P0fd806ZNbn7rrbe6eVQX\n0TGaMWNGRRa9Pu3bt8/No5qLanfjxo1uHm2j972Jvl/bt2/P9ZjRqhGjRo1yc29ViuOOO84dW83V\nVHhnGAAAAIXFZBgAAACFxWQYAAAAhcVkGAAAAIXV6WTYzG4ws01m9kJZdpWZrTezxdm/P6ntZgIA\nAADV15XW6hslfUfSDw/Kv5VS+nrVtwi53HPPPW7udaVecskl7ljv2umonjydrXm7YPOsBFCt+4jy\nvXv3uvkjjzzi5vfff7+bd3R0VGSzZs1yx77nPe9x82HDhrl5tOoBKnl1kXe1k+g4RCvVrFy50s1f\nffXVisxbdUSKj/HOnTvdfMOGDW6+detWN4868r189+7d7tjouRJ18EcrZFTjed6IvO9LtKrBQw89\n5OZPPPGEm0d1MXbsWDc/9thjK7KoRl9//XU3X7ZsmZu/9NJLbu69hkrxz/qjjz66IotWfIlWcdm1\na1euvL293c29ba/Gz7nOdPqTIaX0iKSuryUDAAAA1ImevE1yhZn9PjuNYkzVtggAAADoJd2dDH9P\n0lRJJ0tqkfSNaKCZXW5mC81sYWtrazcfDuh91C7qEXWLekXtoq90azKcUtqYUtqfUnpT0vclnfY2\nY69LKTWnlJrHjRvX3e0Eeh21i3pE3aJeUbvoK92aDJvZxLJPL5bkX58QAAAA6Mc6XU3CzH4i6VxJ\nTWb2qqR/kHSumZ0sKUlaI+nTNdxGKL5m+R133OHmQ4cOrci+9rWvuWO5vn119EW3dt5u2mp030Yd\n71Fn8/XXX+/mLS0tbn7kkUdWZB/84Afdscccc4ybDx482M3RM3lrfMiQIW7uHWNJOuWUU9zce43a\ntGlTrm2JVraIVmqIVh9YtWqVm3td8HlWnqi16Llfz6tMRPvkHYstW7a4Y5999lk3j07TiFYCiV5z\nvOfAtm3b3LFtbW1uHtVctFJD9Lo4Y8YMNz/ssMMqsqhGo1VZtm/f7ubRMRo1apSbT5w4sSKLnqPV\nrN1OJ8MppflO7P90AwAAAOoIi24CAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsDptoEP/EHXkP/ro\no27+kY98pCKLrksOeKJu4s2b/auzf/e733Xzp556ys2jlQYuvPDCiuy9732vO/bQQw918wED/N/z\n67lzvh5FK9WMHDnSzU844QQ3Hzt2bEUWrQ6Qd1ui7vj77rvPzZ955hk397rmo475qPajuqWe8/Fe\nu6IVHKKVmqJ6ifJo1QRv1ZxodYhoNYnXX3/dzaPXv2iN5mi8tyrFY4895o6NXs+j55H33JWkmTNn\nuvmUKVMqst5YHYh3hgEAAFBYTIYBAABQWEyGAQAAUFhMhgEAAFBYNND1M4sXL3bzK664ws29yyhK\n0le/+tWqbRPqT57GmqhRLrrsaNRYdPfdd7v57t273fz000938wULFlRkUUNIdJlO9K6o3qLGr6gh\nJmqs8xp/JkyYUJVtaW9vd/OVK1e6edQQ5DXFRdsYXRo6as6qRqNckZrtvH2NmhabmprcPHrNiV4X\nowY9rxEt76XEd+zY4ebRPkWv6dHjrl69uiL7+c9/7o7duHGjm0c1PXv2bDe/5JJL3Nw7Hr1xOWbe\nGQYAAEBhMRkGAABAYTEZBgAAQGExGQYAAEBhMRkGAABAYdGK3Ue8yx9K0vz58918//79bv7Rj37U\nzbn0cjFE3bTRpUE9UefxunXr3PyGG25w8+iSoVH3/WWXXebmxx13XEXWG93EqL68KztE3fHe8c9T\n4283vqOjw82jFS+iSyx7+dFHH+2OPeSQQ9y8WvVc9OeFtypHtGpEc3Ozmy9btszNo8vRR69/ixYt\nqsiWLl3qjo1WE4lqNO8KGVHtepc237p1qzt20qRJbh6tGnH55Ze7+YwZM9x86NChFVn0elFNvDMM\nAACAwmIyDAAAgMJiMgwAAIDCYjIMAACAwmIyDAAAgMLqdDUJMzta0g8ljZeUJF2XUrrWzMZK+pmk\nKZLWSPpwSqmyJbHgok79efPmufny5cvdfPr06W7+la98pXsbhkLyOup37tzpjr3lllvcfPHixW4e\ndfyed955uXKvQ7ro3fGNJu/x9Gor72oS0Wtx1Km/Y8cONx82bJibH3744RXZmDFjurh16I6ojrxV\nGYYPH+6OnTVrlpuvX7/ezbdv3+7mzz77rJt7KzVEr7l5azpaTSK6H2+lBkk64YQTKrJzzjnHHTtz\n5kw3j8ZPnjw517Z4z/XeeP3vyjvDHZK+kFI6SdJsSZ8xs5MkXSnpwZTSCZIezD4HAAAA6kank+GU\nUktK6dns43ZJyyRNknSRpJuyYTdJ+kCtNhIAAACohVznDJvZFEmnSHpa0viUUkt20waVTqPwvuZy\nM1toZgtbW1t7sKlA76J2UY+oW9Qrahd9pcuTYTMbIek2SZ9LKW0rvy2VTlBxT1JJKV2XUmpOKTWP\nGzeuRxsL9CZqF/WIukW9onbRV7o0GTazwSpNhH+cUro9izea2cTs9omSNtVmEwEAAIDa6MpqEibp\neknLUkrfLLvpbkkLJF2T/X9XTbawzkXXMX/44Ydz3c+PfvQjNx87dmzeTUIBRN3E+/btq8hefPFF\nd+xvfvMbN4+68qdMmeLmn/zkJ9086rSv5XXoo+8Lq1X0b3mOT3SMo3zPnj1uHq0yEdXtoYceWpHt\n3r3bHes9D6X8qwnA59XLoEH+dKepqcnNL730Ujd/5zvf6eYPPfSQm7/wwgsV2Zo1a9yx0XwhqosJ\nEya4+SmnnOLmZ5xxhptPmzatIhs9erQ7dtSoUW7u1b8Uf9+j53RfvRZ3OhmWdKakv5D0vJkdWFPp\nyypNgm8xs09KWivpw7XZRAAAAKA2Op0Mp5QekxRN1edUd3MAAACA3sMV6AAAAFBYTIYBAABQWEyG\nAQAAUFhdaaBDF7zxxhtuPnv27Fz3c/PNN7t51B2KYou6jPfv3+/mW7Zsqcgef/xxd2zU2Tx+vHt9\nHc2fP9/No+7rwYMHu7nXTRx1GOftvq/G/bDyRO3kOT55j0Pe50q0asrOnTvdfO/evRVZW1ubOzZa\nwSLvShjUYtdF36uBAwe6+YgRI9z8Xe96l5tHP6O9lUOi1USiPKrRaKWGaGWHaF+9FXyiVX2i72Pe\n8f0N7wwDAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsJgMAwAAoLBYTaJKfvCDH7j5qlWrct3PWWed\n5eb10pGJ2oi6yaOO96hb/bXXXqvI1q9f745tampy85kzZ7r5vHnz3DzqbM5T03lXjch7Pzy/+jfv\n+ORdeSHqyG9vb3fz1tZWN9+2bZubjxw5ssuPGeXVqnP0XPSaEOVDhgzJlXvq+fjX+2so7wwDAACg\nsJgMAwAAoLCYDAMAAKCwmAwDAACgsJgMAwAAoLBYTaIbVqxYUZFdddVVvb8hqAt5OoTzrhrR0dHh\n5lHH+8qVKyuyXbt2uWOPOOIIN49Wk5gwYYKbR13GfbGyQ713PKNzUV1Fz5V9+/a5+dChQ9382GOP\ndfMxY8ZUZFOnTnXHjhgxws0HDPDfn6Jui4Hj3Hd4ZxgAAACFxWQYAAAAhcVkGAAAAIXFZBgAAACF\n1WkDnZkdLemHksZLSpKuSylda2ZXSfpLSQeuWfnllNK9tdrQ/uTRRx+tyKKGpcj06dPdPLp0LepX\nNS4lGzX/7Ny5082jSyyvXbu2Itu9e7c7dtiwYW4eNcoNGtT7/bi1bjihoaVxRPUZ1XN0efF3v/vd\nbu41xU2u/Qu7AAAEuElEQVSaNMkdO3r0aDePtjHvpYEB5NOVn14dkr6QUnrWzEZKWmRmD2S3fSul\n9PXabR4AAABQO51OhlNKLZJaso/bzWyZJP/XXQAAAKCO5Dpn2MymSDpF0tNZdIWZ/d7MbjCzykUW\nS19zuZktNLOFra2t3hCgX6J2UY+oW9Qrahd9pcuTYTMbIek2SZ9LKW2T9D1JUyWdrNI7x9/wvi6l\ndF1KqTml1Dxu3LgqbDLQO6hd1CPqFvWK2kVf6dJk2MwGqzQR/nFK6XZJSiltTCntTym9Ken7kk6r\n3WYCAAAA1deV1SRM0vWSlqWUvlmWT8zOJ5akiyW9UJtNrG9nnHGGmz/wwANuzmoSxRZ1h0eXaR08\neLCbH3744W5+0kknVWTRpWG9y8tK0jve8Q43HzlypJvn7Xj3VtSo9SWd6cpvHNFzZciQIW4e1Xm0\n4kNUcwMHDuxS9nY5q0YAfaMrq0mcKekvJD1vZouz7MuS5pvZySott7ZG0qdrsoUAAABAjXRlNYnH\nJHm/lhZiTWEAAAA0Lq5ABwAAgMJiMgwAAIDCYjIMAACAwupKAx0O8olPfKJLGRDJ2zUerRoR5cOH\nD3fzyZMnV2RRd3yU5xV19+fpkKfLHgfLWxNRHq3sUK3tAdD/8c4wAAAACovJMAAAAAqLyTAAAAAK\ni8kwAAAACovJMAAAAArLqtUx3qUHM2uVtLbXHrD2miS19fVG9JL+tK+TU0rjevMBqd261Z/2k7qt\njv50TGupP+0ntdtz/el41lp/2tcu1W6vToYbjZktTCk19/V29IYi7WsRFOV4FmU/i6Qox7Qo+1kU\nRTqe9bivnCYBAACAwmIyDAAAgMJiMtwz1/X1BvSiIu1rERTleBZlP4ukKMe0KPtZFEU6nnW3r5wz\nDAAAgMLinWEAAAAUFpPhLjKzG8xsk5m9UJaNNbMHzGxF9v+YvtzGajCzo83sITNbamZLzOyzWd5w\n+1oU1G7j7WsRULeNt69FQe3W374yGe66GyVdcFB2paQHU0onSHow+7zedUj6QkrpJEmzJX3GzE5S\nY+5rUdwoarfR9rUIbhR122j7WhQ3itqtq31lMtxFKaVHJG0+KL5I0k3ZxzdJ+kCvblQNpJRaUkrP\nZh+3S1omaZIacF+LgtptvH0tAuq28fa1KKjd+ttXJsM9Mz6l1JJ9vEHS+L7cmGozsymSTpH0tBp8\nXwuooY8ntduwGvpYUrcNraGPZ73XLpPhKkmlZTkaZmkOMxsh6TZJn0spbSu/rdH2tega7XhSu8XQ\naMeSui2ORjuejVC7TIZ7ZqOZTZSk7P9Nfbw9VWFmg1Uq7B+nlG7P4obc1wJryONJ7Ta8hjyW1G0h\nNOTxbJTaZTLcM3dLWpB9vEDSXX24LVVhZibpeknLUkrfLLup4fa14BrueFK7hdBwx5K6LYyGO56N\nVLtcdKOLzOwnks6V1CRpo6R/kHSnpFskHSNpraQPp5QOPmm+rpjZWZIelfS8pDez+MsqnQfUUPta\nFNQutVuPqFvqtl5Ru/VXu0yGAQAAUFicJgEAAIDCYjIMAACAwmIyDAAAgMJiMgwAAIDCYjIMAACA\nwmIyDAAAgMJiMgwAAIDCYjIMAACAwvr/nULp1p8Yt+AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x141020f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# learning images on the test set\n",
    "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))\n",
    "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax1.set_title('True image')\n",
    "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax2.set_title('Learned image')\n",
    "ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax3.set_title('Learned image')\n",
    "ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax4.set_title('Learned image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('loss', 139.73684648437501)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the ELBO which is minus the loss for test set\n",
    "metric = mx.metric.Loss()\n",
    "model.score(nd_iter_test, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. All together: MXNet-based class VAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from VAE import VAE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can directly call the class `VAE` to do the training:\n",
    "\n",
    "```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,\n",
    "batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,\n",
    "initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```\n",
    "\n",
    "The outputs are the learned model and training loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Train-loss=377.146422\n",
      "INFO:root:Epoch[0] Time cost=5.989\n",
      "INFO:root:Epoch[1] Train-loss=211.998043\n",
      "INFO:root:Epoch[1] Time cost=6.303\n",
      "INFO:root:Epoch[2] Train-loss=207.103096\n",
      "INFO:root:Epoch[2] Time cost=7.368\n",
      "INFO:root:Epoch[3] Train-loss=204.958183\n",
      "INFO:root:Epoch[3] Time cost=7.530\n",
      "INFO:root:Epoch[4] Train-loss=203.342700\n",
      "INFO:root:Epoch[4] Time cost=8.887\n",
      "INFO:root:Epoch[5] Train-loss=201.649251\n",
      "INFO:root:Epoch[5] Time cost=9.147\n",
      "INFO:root:Epoch[6] Train-loss=199.782661\n",
      "INFO:root:Epoch[6] Time cost=8.924\n",
      "INFO:root:Epoch[7] Train-loss=198.044015\n",
      "INFO:root:Epoch[7] Time cost=8.920\n",
      "INFO:root:Epoch[8] Train-loss=195.732077\n",
      "INFO:root:Epoch[8] Time cost=8.857\n",
      "INFO:root:Epoch[9] Train-loss=194.070547\n",
      "INFO:root:Epoch[9] Time cost=9.216\n",
      "INFO:root:Epoch[10] Train-loss=193.186871\n",
      "INFO:root:Epoch[10] Time cost=8.966\n",
      "INFO:root:Epoch[11] Train-loss=192.700208\n",
      "INFO:root:Epoch[11] Time cost=8.843\n",
      "INFO:root:Epoch[12] Train-loss=192.191504\n",
      "INFO:root:Epoch[12] Time cost=8.152\n",
      "INFO:root:Epoch[13] Train-loss=191.842837\n",
      "INFO:root:Epoch[13] Time cost=6.180\n",
      "INFO:root:Epoch[14] Train-loss=191.310450\n",
      "INFO:root:Epoch[14] Time cost=6.067\n",
      "INFO:root:Epoch[15] Train-loss=190.520681\n",
      "INFO:root:Epoch[15] Time cost=6.058\n",
      "INFO:root:Epoch[16] Train-loss=189.784146\n",
      "INFO:root:Epoch[16] Time cost=6.046\n",
      "INFO:root:Epoch[17] Train-loss=188.515020\n",
      "INFO:root:Epoch[17] Time cost=6.062\n",
      "INFO:root:Epoch[18] Train-loss=187.530712\n",
      "INFO:root:Epoch[18] Time cost=6.088\n",
      "INFO:root:Epoch[19] Train-loss=186.194826\n",
      "INFO:root:Epoch[19] Time cost=6.491\n",
      "INFO:root:Epoch[20] Train-loss=185.492288\n",
      "INFO:root:Epoch[20] Time cost=6.182\n",
      "INFO:root:Epoch[21] Train-loss=184.922654\n",
      "INFO:root:Epoch[21] Time cost=6.058\n",
      "INFO:root:Epoch[22] Train-loss=184.677911\n",
      "INFO:root:Epoch[22] Time cost=6.042\n",
      "INFO:root:Epoch[23] Train-loss=183.921396\n",
      "INFO:root:Epoch[23] Time cost=5.994\n",
      "INFO:root:Epoch[24] Train-loss=183.600690\n",
      "INFO:root:Epoch[24] Time cost=6.038\n",
      "INFO:root:Epoch[25] Train-loss=183.388476\n",
      "INFO:root:Epoch[25] Time cost=6.025\n",
      "INFO:root:Epoch[26] Train-loss=182.972208\n",
      "INFO:root:Epoch[26] Time cost=6.014\n",
      "INFO:root:Epoch[27] Train-loss=182.561678\n",
      "INFO:root:Epoch[27] Time cost=6.064\n",
      "INFO:root:Epoch[28] Train-loss=182.475261\n",
      "INFO:root:Epoch[28] Time cost=5.983\n",
      "INFO:root:Epoch[29] Train-loss=182.308808\n",
      "INFO:root:Epoch[29] Time cost=6.371\n",
      "INFO:root:Epoch[30] Train-loss=182.135900\n",
      "INFO:root:Epoch[30] Time cost=6.038\n",
      "INFO:root:Epoch[31] Train-loss=181.978367\n",
      "INFO:root:Epoch[31] Time cost=6.924\n",
      "INFO:root:Epoch[32] Train-loss=181.677153\n",
      "INFO:root:Epoch[32] Time cost=8.205\n",
      "INFO:root:Epoch[33] Train-loss=181.677775\n",
      "INFO:root:Epoch[33] Time cost=6.017\n",
      "INFO:root:Epoch[34] Train-loss=181.257998\n",
      "INFO:root:Epoch[34] Time cost=6.056\n",
      "INFO:root:Epoch[35] Train-loss=181.125288\n",
      "INFO:root:Epoch[35] Time cost=6.020\n",
      "INFO:root:Epoch[36] Train-loss=181.018858\n",
      "INFO:root:Epoch[36] Time cost=6.035\n",
      "INFO:root:Epoch[37] Train-loss=180.785110\n",
      "INFO:root:Epoch[37] Time cost=6.049\n",
      "INFO:root:Epoch[38] Train-loss=180.452598\n",
      "INFO:root:Epoch[38] Time cost=6.083\n",
      "INFO:root:Epoch[39] Train-loss=180.362733\n",
      "INFO:root:Epoch[39] Time cost=6.198\n",
      "INFO:root:Epoch[40] Train-loss=180.060788\n",
      "INFO:root:Epoch[40] Time cost=6.049\n",
      "INFO:root:Epoch[41] Train-loss=180.022728\n",
      "INFO:root:Epoch[41] Time cost=6.135\n",
      "INFO:root:Epoch[42] Train-loss=179.648499\n",
      "INFO:root:Epoch[42] Time cost=6.055\n",
      "INFO:root:Epoch[43] Train-loss=179.507952\n",
      "INFO:root:Epoch[43] Time cost=6.108\n",
      "INFO:root:Epoch[44] Train-loss=179.303132\n",
      "INFO:root:Epoch[44] Time cost=6.020\n",
      "INFO:root:Epoch[45] Train-loss=178.945211\n",
      "INFO:root:Epoch[45] Time cost=6.004\n",
      "INFO:root:Epoch[46] Train-loss=178.808598\n",
      "INFO:root:Epoch[46] Time cost=6.016\n",
      "INFO:root:Epoch[47] Train-loss=178.550906\n",
      "INFO:root:Epoch[47] Time cost=6.050\n",
      "INFO:root:Epoch[48] Train-loss=178.403674\n",
      "INFO:root:Epoch[48] Time cost=6.115\n",
      "INFO:root:Epoch[49] Train-loss=178.237544\n",
      "INFO:root:Epoch[49] Time cost=6.004\n",
      "INFO:root:Epoch[50] Train-loss=178.033747\n",
      "INFO:root:Epoch[50] Time cost=6.051\n",
      "INFO:root:Epoch[51] Train-loss=177.802884\n",
      "INFO:root:Epoch[51] Time cost=6.028\n",
      "INFO:root:Epoch[52] Train-loss=177.533980\n",
      "INFO:root:Epoch[52] Time cost=6.052\n",
      "INFO:root:Epoch[53] Train-loss=177.490143\n",
      "INFO:root:Epoch[53] Time cost=6.019\n",
      "INFO:root:Epoch[54] Train-loss=177.136637\n",
      "INFO:root:Epoch[54] Time cost=6.014\n",
      "INFO:root:Epoch[55] Train-loss=177.062524\n",
      "INFO:root:Epoch[55] Time cost=6.024\n",
      "INFO:root:Epoch[56] Train-loss=176.869033\n",
      "INFO:root:Epoch[56] Time cost=6.065\n",
      "INFO:root:Epoch[57] Train-loss=176.704606\n",
      "INFO:root:Epoch[57] Time cost=6.037\n",
      "INFO:root:Epoch[58] Train-loss=176.470091\n",
      "INFO:root:Epoch[58] Time cost=6.012\n",
      "INFO:root:Epoch[59] Train-loss=176.261440\n",
      "INFO:root:Epoch[59] Time cost=6.215\n",
      "INFO:root:Epoch[60] Train-loss=176.133904\n",
      "INFO:root:Epoch[60] Time cost=6.042\n",
      "INFO:root:Epoch[61] Train-loss=175.941920\n",
      "INFO:root:Epoch[61] Time cost=6.000\n",
      "INFO:root:Epoch[62] Train-loss=175.731296\n",
      "INFO:root:Epoch[62] Time cost=6.025\n",
      "INFO:root:Epoch[63] Train-loss=175.613303\n",
      "INFO:root:Epoch[63] Time cost=6.002\n",
      "INFO:root:Epoch[64] Train-loss=175.438844\n",
      "INFO:root:Epoch[64] Time cost=5.982\n",
      "INFO:root:Epoch[65] Train-loss=175.254716\n",
      "INFO:root:Epoch[65] Time cost=6.016\n",
      "INFO:root:Epoch[66] Train-loss=175.090210\n",
      "INFO:root:Epoch[66] Time cost=6.008\n",
      "INFO:root:Epoch[67] Train-loss=174.895443\n",
      "INFO:root:Epoch[67] Time cost=6.008\n",
      "INFO:root:Epoch[68] Train-loss=174.701321\n",
      "INFO:root:Epoch[68] Time cost=6.418\n",
      "INFO:root:Epoch[69] Train-loss=174.553292\n",
      "INFO:root:Epoch[69] Time cost=6.072\n",
      "INFO:root:Epoch[70] Train-loss=174.349379\n",
      "INFO:root:Epoch[70] Time cost=6.048\n",
      "INFO:root:Epoch[71] Train-loss=174.174641\n",
      "INFO:root:Epoch[71] Time cost=6.036\n",
      "INFO:root:Epoch[72] Train-loss=173.966333\n",
      "INFO:root:Epoch[72] Time cost=6.017\n",
      "INFO:root:Epoch[73] Train-loss=173.798454\n",
      "INFO:root:Epoch[73] Time cost=6.018\n",
      "INFO:root:Epoch[74] Train-loss=173.635657\n",
      "INFO:root:Epoch[74] Time cost=5.985\n",
      "INFO:root:Epoch[75] Train-loss=173.423795\n",
      "INFO:root:Epoch[75] Time cost=6.016\n",
      "INFO:root:Epoch[76] Train-loss=173.273981\n",
      "INFO:root:Epoch[76] Time cost=6.018\n",
      "INFO:root:Epoch[77] Train-loss=173.073401\n",
      "INFO:root:Epoch[77] Time cost=5.996\n",
      "INFO:root:Epoch[78] Train-loss=172.888044\n",
      "INFO:root:Epoch[78] Time cost=6.035\n",
      "INFO:root:Epoch[79] Train-loss=172.694943\n",
      "INFO:root:Epoch[79] Time cost=8.492\n",
      "INFO:root:Epoch[80] Train-loss=172.504260\n",
      "INFO:root:Epoch[80] Time cost=7.380\n",
      "INFO:root:Epoch[81] Train-loss=172.323245\n",
      "INFO:root:Epoch[81] Time cost=6.063\n",
      "INFO:root:Epoch[82] Train-loss=172.131274\n",
      "INFO:root:Epoch[82] Time cost=6.209\n",
      "INFO:root:Epoch[83] Train-loss=171.932986\n",
      "INFO:root:Epoch[83] Time cost=6.060\n",
      "INFO:root:Epoch[84] Train-loss=171.755262\n",
      "INFO:root:Epoch[84] Time cost=6.068\n",
      "INFO:root:Epoch[85] Train-loss=171.556803\n",
      "INFO:root:Epoch[85] Time cost=6.004\n",
      "INFO:root:Epoch[86] Train-loss=171.384773\n",
      "INFO:root:Epoch[86] Time cost=6.059\n",
      "INFO:root:Epoch[87] Train-loss=171.185034\n",
      "INFO:root:Epoch[87] Time cost=6.001\n",
      "INFO:root:Epoch[88] Train-loss=170.995980\n",
      "INFO:root:Epoch[88] Time cost=6.143\n",
      "INFO:root:Epoch[89] Train-loss=170.818701\n",
      "INFO:root:Epoch[89] Time cost=6.690\n",
      "INFO:root:Epoch[90] Train-loss=170.629929\n",
      "INFO:root:Epoch[90] Time cost=6.869\n",
      "INFO:root:Epoch[91] Train-loss=170.450824\n",
      "INFO:root:Epoch[91] Time cost=7.156\n",
      "INFO:root:Epoch[92] Train-loss=170.261806\n",
      "INFO:root:Epoch[92] Time cost=6.972\n",
      "INFO:root:Epoch[93] Train-loss=170.070318\n",
      "INFO:root:Epoch[93] Time cost=6.595\n",
      "INFO:root:Epoch[94] Train-loss=169.906993\n",
      "INFO:root:Epoch[94] Time cost=6.561\n",
      "INFO:root:Epoch[95] Train-loss=169.734455\n",
      "INFO:root:Epoch[95] Time cost=6.744\n",
      "INFO:root:Epoch[96] Train-loss=169.564318\n",
      "INFO:root:Epoch[96] Time cost=6.601\n",
      "INFO:root:Epoch[97] Train-loss=169.373926\n",
      "INFO:root:Epoch[97] Time cost=6.725\n",
      "INFO:root:Epoch[98] Train-loss=169.215408\n",
      "INFO:root:Epoch[98] Time cost=6.391\n",
      "INFO:root:Epoch[99] Train-loss=169.039854\n",
      "INFO:root:Epoch[99] Time cost=6.677\n",
      "INFO:root:Epoch[100] Train-loss=168.869222\n",
      "INFO:root:Epoch[100] Time cost=6.370\n",
      "INFO:root:Epoch[101] Train-loss=168.703175\n",
      "INFO:root:Epoch[101] Time cost=6.607\n",
      "INFO:root:Epoch[102] Train-loss=168.523054\n",
      "INFO:root:Epoch[102] Time cost=6.368\n",
      "INFO:root:Epoch[103] Train-loss=168.365964\n",
      "INFO:root:Epoch[103] Time cost=10.267\n",
      "INFO:root:Epoch[104] Train-loss=168.181174\n",
      "INFO:root:Epoch[104] Time cost=11.132\n",
      "INFO:root:Epoch[105] Train-loss=168.021498\n",
      "INFO:root:Epoch[105] Time cost=10.187\n",
      "INFO:root:Epoch[106] Train-loss=167.858251\n",
      "INFO:root:Epoch[106] Time cost=10.676\n",
      "INFO:root:Epoch[107] Train-loss=167.690670\n",
      "INFO:root:Epoch[107] Time cost=10.973\n",
      "INFO:root:Epoch[108] Train-loss=167.535069\n",
      "INFO:root:Epoch[108] Time cost=10.108\n",
      "INFO:root:Epoch[109] Train-loss=167.373971\n",
      "INFO:root:Epoch[109] Time cost=11.013\n",
      "INFO:root:Epoch[110] Train-loss=167.207507\n",
      "INFO:root:Epoch[110] Time cost=11.427\n",
      "INFO:root:Epoch[111] Train-loss=167.043077\n",
      "INFO:root:Epoch[111] Time cost=10.349\n",
      "INFO:root:Epoch[112] Train-loss=166.884060\n",
      "INFO:root:Epoch[112] Time cost=13.129\n",
      "INFO:root:Epoch[113] Train-loss=166.746976\n",
      "INFO:root:Epoch[113] Time cost=11.255\n",
      "INFO:root:Epoch[114] Train-loss=166.572499\n",
      "INFO:root:Epoch[114] Time cost=10.037\n",
      "INFO:root:Epoch[115] Train-loss=166.445170\n",
      "INFO:root:Epoch[115] Time cost=10.406\n",
      "INFO:root:Epoch[116] Train-loss=166.284912\n",
      "INFO:root:Epoch[116] Time cost=10.170\n",
      "INFO:root:Epoch[117] Train-loss=166.171475\n",
      "INFO:root:Epoch[117] Time cost=10.034\n",
      "INFO:root:Epoch[118] Train-loss=166.015457\n",
      "INFO:root:Epoch[118] Time cost=10.047\n",
      "INFO:root:Epoch[119] Train-loss=165.882208\n",
      "INFO:root:Epoch[119] Time cost=10.008\n",
      "INFO:root:Epoch[120] Train-loss=165.753836\n",
      "INFO:root:Epoch[120] Time cost=10.056\n",
      "INFO:root:Epoch[121] Train-loss=165.626045\n",
      "INFO:root:Epoch[121] Time cost=10.704\n",
      "INFO:root:Epoch[122] Train-loss=165.492859\n",
      "INFO:root:Epoch[122] Time cost=10.609\n",
      "INFO:root:Epoch[123] Train-loss=165.361132\n",
      "INFO:root:Epoch[123] Time cost=10.027\n",
      "INFO:root:Epoch[124] Train-loss=165.256487\n",
      "INFO:root:Epoch[124] Time cost=11.225\n",
      "INFO:root:Epoch[125] Train-loss=165.119995\n",
      "INFO:root:Epoch[125] Time cost=11.266\n",
      "INFO:root:Epoch[126] Train-loss=165.012773\n",
      "INFO:root:Epoch[126] Time cost=10.547\n",
      "INFO:root:Epoch[127] Train-loss=164.898748\n",
      "INFO:root:Epoch[127] Time cost=10.339\n",
      "INFO:root:Epoch[128] Train-loss=164.775702\n",
      "INFO:root:Epoch[128] Time cost=10.875\n",
      "INFO:root:Epoch[129] Train-loss=164.692449\n",
      "INFO:root:Epoch[129] Time cost=8.412\n",
      "INFO:root:Epoch[130] Train-loss=164.564323\n",
      "INFO:root:Epoch[130] Time cost=7.239\n",
      "INFO:root:Epoch[131] Train-loss=164.468273\n",
      "INFO:root:Epoch[131] Time cost=10.096\n",
      "INFO:root:Epoch[132] Train-loss=164.328320\n",
      "INFO:root:Epoch[132] Time cost=9.680\n",
      "INFO:root:Epoch[133] Train-loss=164.256156\n",
      "INFO:root:Epoch[133] Time cost=10.707\n",
      "INFO:root:Epoch[134] Train-loss=164.151625\n",
      "INFO:root:Epoch[134] Time cost=13.835\n",
      "INFO:root:Epoch[135] Train-loss=164.046402\n",
      "INFO:root:Epoch[135] Time cost=10.049\n",
      "INFO:root:Epoch[136] Train-loss=163.960676\n",
      "INFO:root:Epoch[136] Time cost=9.625\n",
      "INFO:root:Epoch[137] Train-loss=163.873193\n",
      "INFO:root:Epoch[137] Time cost=9.845\n",
      "INFO:root:Epoch[138] Train-loss=163.783837\n",
      "INFO:root:Epoch[138] Time cost=9.618\n",
      "INFO:root:Epoch[139] Train-loss=163.658903\n",
      "INFO:root:Epoch[139] Time cost=10.411\n",
      "INFO:root:Epoch[140] Train-loss=163.588920\n",
      "INFO:root:Epoch[140] Time cost=9.633\n",
      "INFO:root:Epoch[141] Train-loss=163.493254\n",
      "INFO:root:Epoch[141] Time cost=10.668\n",
      "INFO:root:Epoch[142] Train-loss=163.401188\n",
      "INFO:root:Epoch[142] Time cost=10.644\n",
      "INFO:root:Epoch[143] Train-loss=163.334470\n",
      "INFO:root:Epoch[143] Time cost=9.665\n",
      "INFO:root:Epoch[144] Train-loss=163.235133\n",
      "INFO:root:Epoch[144] Time cost=9.612\n",
      "INFO:root:Epoch[145] Train-loss=163.168029\n",
      "INFO:root:Epoch[145] Time cost=9.578\n",
      "INFO:root:Epoch[146] Train-loss=163.092392\n",
      "INFO:root:Epoch[146] Time cost=10.215\n",
      "INFO:root:Epoch[147] Train-loss=163.014362\n",
      "INFO:root:Epoch[147] Time cost=12.296\n",
      "INFO:root:Epoch[148] Train-loss=162.891574\n",
      "INFO:root:Epoch[148] Time cost=9.578\n",
      "INFO:root:Epoch[149] Train-loss=162.831664\n",
      "INFO:root:Epoch[149] Time cost=9.536\n",
      "INFO:root:Epoch[150] Train-loss=162.768784\n",
      "INFO:root:Epoch[150] Time cost=9.607\n",
      "INFO:root:Epoch[151] Train-loss=162.695416\n",
      "INFO:root:Epoch[151] Time cost=9.681\n",
      "INFO:root:Epoch[152] Train-loss=162.620814\n",
      "INFO:root:Epoch[152] Time cost=9.464\n",
      "INFO:root:Epoch[153] Train-loss=162.527031\n",
      "INFO:root:Epoch[153] Time cost=9.518\n",
      "INFO:root:Epoch[154] Train-loss=162.466575\n",
      "INFO:root:Epoch[154] Time cost=9.562\n",
      "INFO:root:Epoch[155] Train-loss=162.409388\n",
      "INFO:root:Epoch[155] Time cost=9.483\n",
      "INFO:root:Epoch[156] Train-loss=162.308957\n",
      "INFO:root:Epoch[156] Time cost=9.545\n",
      "INFO:root:Epoch[157] Train-loss=162.211725\n",
      "INFO:root:Epoch[157] Time cost=9.542\n",
      "INFO:root:Epoch[158] Train-loss=162.141098\n",
      "INFO:root:Epoch[158] Time cost=9.768\n",
      "INFO:root:Epoch[159] Train-loss=162.124311\n",
      "INFO:root:Epoch[159] Time cost=7.155\n",
      "INFO:root:Epoch[160] Train-loss=162.013039\n",
      "INFO:root:Epoch[160] Time cost=6.147\n",
      "INFO:root:Epoch[161] Train-loss=161.954485\n",
      "INFO:root:Epoch[161] Time cost=9.121\n",
      "INFO:root:Epoch[162] Train-loss=161.913859\n",
      "INFO:root:Epoch[162] Time cost=9.936\n",
      "INFO:root:Epoch[163] Train-loss=161.830799\n",
      "INFO:root:Epoch[163] Time cost=8.612\n",
      "INFO:root:Epoch[164] Train-loss=161.768672\n",
      "INFO:root:Epoch[164] Time cost=9.722\n",
      "INFO:root:Epoch[165] Train-loss=161.689120\n",
      "INFO:root:Epoch[165] Time cost=9.478\n",
      "INFO:root:Epoch[166] Train-loss=161.598279\n",
      "INFO:root:Epoch[166] Time cost=9.466\n",
      "INFO:root:Epoch[167] Train-loss=161.551172\n",
      "INFO:root:Epoch[167] Time cost=9.419\n",
      "INFO:root:Epoch[168] Train-loss=161.488880\n",
      "INFO:root:Epoch[168] Time cost=9.457\n",
      "INFO:root:Epoch[169] Train-loss=161.410458\n",
      "INFO:root:Epoch[169] Time cost=9.504\n",
      "INFO:root:Epoch[170] Train-loss=161.340681\n",
      "INFO:root:Epoch[170] Time cost=9.866\n",
      "INFO:root:Epoch[171] Train-loss=161.281700\n",
      "INFO:root:Epoch[171] Time cost=9.526\n",
      "INFO:root:Epoch[172] Train-loss=161.215523\n",
      "INFO:root:Epoch[172] Time cost=9.511\n",
      "INFO:root:Epoch[173] Train-loss=161.152452\n",
      "INFO:root:Epoch[173] Time cost=9.498\n",
      "INFO:root:Epoch[174] Train-loss=161.058544\n",
      "INFO:root:Epoch[174] Time cost=9.561\n",
      "INFO:root:Epoch[175] Train-loss=161.036475\n",
      "INFO:root:Epoch[175] Time cost=9.463\n",
      "INFO:root:Epoch[176] Train-loss=161.009996\n",
      "INFO:root:Epoch[176] Time cost=9.629\n",
      "INFO:root:Epoch[177] Train-loss=160.853546\n",
      "INFO:root:Epoch[177] Time cost=9.518\n",
      "INFO:root:Epoch[178] Train-loss=160.860520\n",
      "INFO:root:Epoch[178] Time cost=9.395\n",
      "INFO:root:Epoch[179] Train-loss=160.810621\n",
      "INFO:root:Epoch[179] Time cost=9.452\n",
      "INFO:root:Epoch[180] Train-loss=160.683071\n",
      "INFO:root:Epoch[180] Time cost=9.411\n",
      "INFO:root:Epoch[181] Train-loss=160.674101\n",
      "INFO:root:Epoch[181] Time cost=8.784\n",
      "INFO:root:Epoch[182] Train-loss=160.554823\n",
      "INFO:root:Epoch[182] Time cost=7.265\n",
      "INFO:root:Epoch[183] Train-loss=160.536528\n",
      "INFO:root:Epoch[183] Time cost=6.108\n",
      "INFO:root:Epoch[184] Train-loss=160.525913\n",
      "INFO:root:Epoch[184] Time cost=6.349\n",
      "INFO:root:Epoch[185] Train-loss=160.399412\n",
      "INFO:root:Epoch[185] Time cost=7.364\n",
      "INFO:root:Epoch[186] Train-loss=160.380027\n",
      "INFO:root:Epoch[186] Time cost=7.651\n",
      "INFO:root:Epoch[187] Train-loss=160.272921\n",
      "INFO:root:Epoch[187] Time cost=7.309\n",
      "INFO:root:Epoch[188] Train-loss=160.243907\n",
      "INFO:root:Epoch[188] Time cost=7.162\n",
      "INFO:root:Epoch[189] Train-loss=160.194351\n",
      "INFO:root:Epoch[189] Time cost=8.941\n",
      "INFO:root:Epoch[190] Train-loss=160.130400\n",
      "INFO:root:Epoch[190] Time cost=10.242\n",
      "INFO:root:Epoch[191] Train-loss=160.073841\n",
      "INFO:root:Epoch[191] Time cost=10.528\n",
      "INFO:root:Epoch[192] Train-loss=160.021623\n",
      "INFO:root:Epoch[192] Time cost=9.482\n",
      "INFO:root:Epoch[193] Train-loss=159.938673\n",
      "INFO:root:Epoch[193] Time cost=9.465\n",
      "INFO:root:Epoch[194] Train-loss=159.885823\n",
      "INFO:root:Epoch[194] Time cost=9.523\n",
      "INFO:root:Epoch[195] Train-loss=159.886516\n",
      "INFO:root:Epoch[195] Time cost=9.599\n",
      "INFO:root:Epoch[196] Train-loss=159.797400\n",
      "INFO:root:Epoch[196] Time cost=8.675\n",
      "INFO:root:Epoch[197] Train-loss=159.705562\n",
      "INFO:root:Epoch[197] Time cost=9.551\n",
      "INFO:root:Epoch[198] Train-loss=159.738354\n",
      "INFO:root:Epoch[198] Time cost=9.919\n",
      "INFO:root:Epoch[199] Train-loss=159.619932\n",
      "INFO:root:Epoch[199] Time cost=10.121\n"
     ]
    }
   ],
   "source": [
    "# can initilize weights and biases with the learned parameters as follows: \n",
    "# init = mx.initializer.Load(params)\n",
    "\n",
    "# call the VAE, output model contains the learned model and training loss\n",
    "out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# encode test images to obtain mu and logvar which are used for sampling\n",
    "[mu,logvar] = VAE.encoder(out,image_test)\n",
    "# sample in the latent space\n",
    "z = VAE.sampler(mu,logvar)\n",
    "# decode from the latent space to obtain reconstructed images\n",
    "x_construction = VAE.decoder(out,z)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADSCAYAAACvmc1VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuU3GWd5/HPl9xJQsilE5JAEgIBQgQCtshNyQpBRVE5\nq86gO6COg7PHdeTozIjumRWdYWX3eJ11jw4uCOIFdcHb6OhGRA0YkA6JQIgxkgSTkPu1cyfJs3/U\nL0yR+n5J/7qrurvqeb/OyUn3p55UP7/+fav6SXU935+llAQAAADk6Li+ngAAAADQV1gMAwAAIFss\nhgEAAJAtFsMAAADIFothAAAAZIvFMAAAALLFYrgfMLP/Y2Yf6+t5AD1hZtPMLJnZwOD2JWY2p5en\nBRwTtYtmRN3WD4thSWa2q+rPYTPbW/X5Oxv99VNK700p/fdGfx00DzNbZWZX9vU86imlNCul9Mu+\nngcai9pFM6Ju8+b+byI3KaURRz42s1WS3ptS+nk03swGppQO9sbcgEahjtGsqF00I+q2/+KV4S4w\ns38ys2+b2bfMrFPSfzKzr5vZLVVjriwW0kc+P9nMvmdmm8xspZm9/yXu/4X7OnI/ZvbR4t8+Z2bX\nmNkbzWy5mW01s7+v+rcXm9kjZrbdzNaZ2T+b2aCq219vZn8wsx1m9r/M7GEze1fV7e81s9+b2TYz\n+zczO6VO3zY0SFELi4tz/hszO7fqtpvN7Bkz6zSzp83s2qrb3lWc/8+Z2RZJtxTZQ2b26aIGVprZ\n66v+zSgzu6OorbXFY2FAcduA4t9tNrMVkt5wjHm/8MqLmd1iZt8tar/TzJ40szOKut9oZqvN7Kqq\nf/tuM1tajF1hZu876r7/vpjjc0VNJzM7vbhtSDHPP5nZBjP7spkN69lZQHdQu9RuM6JuW79uWQx3\n3bWSvilplKRvv9RAMztO0r9KekzSZElzJf2dmV3Rxa91sirnZpKkf5R0h6Q/l3S+pDmSPmlmU4qx\nByV9UNI4SZdKep2k9xXzGC/pO5L+rrh9paQLq+b5H4vb3iypTdKjxTGinzKz8yXdqco5HivpXyT9\n0MyGFEOekfQqVer0E5K+bmYTq+7ilZJWSJog6daqbJkqNfI/Jd1hZlbcdpcqNXa6KvV3laT3Frf9\nlaQ3Fnm7pLeWPJxrJN0jabSkRZJ+pkrdT5b0yeLYjthYfK0TJL1b0ufM7ILie/I6SR+SdGUxzzlH\nfZ3bJJ0haXZx+2RJ/63kXNFD1C6124yo20zqNqXEn6o/klZJuvKo7J8k/eKo7OuSbqn6/EpJq4qP\nL5W04qjx/yDpK8HXfOG+ivvZJWlA8floSUnSy6vG/07SG4P7+ltJ3y0+fo+k+VW3maR1kt5VfD5P\n0g1Vtw+UtF/S5L4+D7n/8eqwyL8k6R+PypZJujy4n8WS3lx8/C5Jfzrq9ndJ+mPV58cX9XaSKk/e\n+yUNq7r9OkkPFh//QtJfV912VfFvBx7rmCTdImle1W3XHFX3I4v7OjG4r+9L+mDx8Z2SPlV12+nF\nvz29qPndkk6ruv1iSSv7+hy36h9ql9ptxj/Ubd51y3uGu251ibFTJU0xs+1V2QBJv+ziv9+cUjpU\nfLy3+HtD1e17JY2QJDM7S9JnJL1clQfVQFVe4ZUqryy/MO+UUjKzNUfN83+b2ReqssOqvDK9totz\nRe+aKukGM/tAVTZYlXMtM7telf+xTytuG6HKqw9HeHW8/sgHKaU9xQsUIySNkTRI0rp/f9FCx1Xd\nx4vqS9KzJY/l6Jr26n6EpO3FrxE/rsqrDcepUutPVs2jo+q+qufUVoxdWHUMpsrjEb2L2qV2mxF1\nm0HdshjuunTU57tVOeFHnFT18WpJy1NKMxs+q8qvNR6R9GcppV1m9req/GpDqrwKXP0eIFPl1xXV\n8/yHlNJLvu0D/cpqSbemlG49+gYzmyrpK5KukLQgpXTIzBar8kR0xNF1fKyvtV/SuORv+lgnqfo9\n5lOcMT1W/DryPknXS/pBSul5M/u+/v241qnyH7gjque0WZUn+VkpJf6D17eoXWq3GVG3GdQt7xnu\nvsWS3mBmo4v3B/1N1W0LJB0wsw+b2dDiTe/nmNnLGzCPkZJ2SNptZjNVvF+48K+SLrDKBryBqry3\nuK3q9i9L+q/Fv5OZnWhmZd+DhMYZVNTPkT8DVXni/Wsze6VVDDezN5jZSEnDVXni3SRVNkBIell3\nv3hKaZ2k/yfpM2Z2gpkdZ2anmdnlxZDvSPobq2wWHS3p5h4c60sZLGmIKsd1sHjF4qqq278j6d1m\nNtPMjlflLUlHjuGwKt+zzxXvoZeZTTaz1zZorqigdiuo3eZC3VZkV7cshrvvLklLVfk1xU8l3Xvk\nhuJ/dFerslltlSr/U/oXVd6IXm8flnSDpM7ia7zwKm9KaYOkP5P0WUlbJJ2mypvm9xe3f7e47btm\ntlPSE5L6dcFm5ieq/A/7yJ9bUkodqmyi+KKkbZL+qMp70JRSelqVt8wsUOXXYedIeriHc7helSfG\np4uv938lHdkc8hVVNmD8TtLjku7v4ddypZQ6VfnP5neKObxD0g+rbv83Sf8s6UFVvh+PFDftL/7+\nyJG8qPOfSzqzEXPFC6hdUbtNiLpVnnVrxZubkQGrtGd5TtJbU0rz+3o+QCMUv+l4StKQ4FeNQL9E\n7aIZtULd8spwizOz1xVvfxiiyq8ynpf02z6eFlBXZnatVXpbjpb0PyT9qFmflJEXahfNqNXqlsVw\n67tMlR6Hm1R5C8S1KaX9L/1PgKbzPlX6Yj4j6ZCk/9y30wG6jNpFM2qpuuVtEgAAAMgWrwwDAAAg\nWz1aDBfvR11mZn80s0a1+AAAAAAaottvkyg6E/xB0lxJayQ9Jum6otWIa9y4cWnatGnd+nrAEQsX\nLtycUmo79sj6oXbRU6tWrdLmzZvt2CPrh7pFPfCci2bV1drtyRXoLlTl+torJMnM7pX0ZlV647mm\nTZumjo6O6GagS8ys7CUoe4zaRU+1t7f3+tekblEPPOeiWXW1dnvyNonJevH1qNfoxZf6BQAAAPq1\nhm+gM7MbzazDzDo2bdrU6C8H1A21i2ZE3aJZUbvoKz1ZDK+VdErV5ycX2YuklG5PKbWnlNrb2nr1\nLUdAj1C7aEbULZoVtYu+0pPF8GOSZpjZqWY2WNKfq+ra1QAAAEB/1+0NdCmlg2b2XyT9TNIASXem\nlJbUbWYAAABAg/Wkm4RSSj+R9JM6zQUAAADoVVyBDgAAANliMQwAAIBssRgGAABAtnr0nmEAAPpC\nSqmh92/Wq1fORotqZJ0ePnzYzQ8ePOjmhw4dcnNvjlH9H3ec/xrqwIH+cjIaH91/Xz3ueGUYAAAA\n2WIxDAAAgGyxGAYAAEC2WAwDAAAgWyyGAQAAkC26SQAAAFQp2wWizPiy9x11gdi3b5+bb9++3c23\nbt3q5gcOHKjJBg8e7I4dNWqUm48ePdrNhw0b5uZluk/0RocJXhkGAABAtlgMAwAAIFsshgEAAJAt\nFsMAAADIFhvoALScel0ClUvy9g+NvvRyPTRyA1WkTH3mXsv12hBXj/uJLqMcbZTbvXu3m69evdrN\nFy1a5OYrVqxwc28j3oknnuiOPf3009181qxZbj5p0iQ3HzFihJsPGjTIzT31rGleGQYAAEC2WAwD\nAAAgWyyGAQAAkC0WwwAAAMgWi2EAAABkq0fdJMxslaROSYckHUwptddjUgBQrdHdBOpx/znv1u+L\nbg+N7shQtpuA1yEg6g5Q9r6jOXqXro3yAQMGlLrvXNSrjsqc06ibxP79+9183bp1bv7oo4+6+WOP\nPebmGzZscHPv0sjjx493x0bdHqLuE0OHDi11P16dRrVb9vHyUurRWu0/pJQ21+F+AAAAgF7F2yQA\nAACQrZ4uhpOkn5vZQjO7sR4TAgAAAHpLTxfDl6WUZkt6vaT3m9mrjx5gZjeaWYeZdWzatKmHXw7o\nPdQumhF1i2ZF7aKv9GgxnFJaW/y9UdL3JF3ojLk9pdSeUmpva2vryZcDehW1i2ZE3aJZUbvoK93e\nQGdmwyUdl1LqLD6+StIn6zYzIEP1uO59d+6nL+bSyLmX2WUvldt9XLb7QLPu1i9zHup1jqNd9l4e\njY06OETjo7mUvR9vfHQfkWjXfNkd+V53gLIdKZq1bqW+6W4S8erl+eefd8dGr4ZH3SGibhIrVqxw\n82HDhrn5mDFjujw2mvvGjRvdPOoycfzxx7v54MGDa7LeqNGedJOYIOl7xWQGSvpmSumndZkVAAAA\n0Au6vRhOKa2QdF4d5wIAAAD0KlqrAQAAIFsshgEAAJAtFsMAAADIVj0ux9zvPPLII27+hS98wc0n\nT57s5tFuyhtuuKEm83ZjvlSO1lNm53y0y/zAgQNuvmfPHjffvXt3qfH79+/vUvZSc9m7d6+bR7uM\no/vp7Ox08127dtVk0Ryj3fSnnXaam8+cOdPNx48f7+bezuZox3+z7r4vU7f16uAQ1Yp37iVp27Zt\nNdnmzZvdsTt27HDzgwcPunnZziNepwbJP9aoPqOfLdHO+9GjR7v5yJEj3TyaO7quXt10vMdGVKNL\nlixx8wULFrj50qVL3TyqizPOOMPNTz311Josqrl9+/a5efQzZ+vWrW4e1foJJ5xQk3nPw/XGIwYA\nAADZYjEMAACAbLEYBgAAQLZYDAMAACBbLIYBAACQrZbsJuF1e5Ck5cuX1+X+b7311pps1KhR7tiL\nLrqoLl+zL0ybNs3NP/rRj7r5lClTGjib/qPM7nvJ300c7chdv369m0e1G12Dfu3atW7uXT8+uqb8\n6tWr3TzaCR0df7QTOOrK4HUaiHYqDxkyxM1f/vKXu/mb3vQmN7/iiivcfOzYsTVZNO/+rmzdet0R\noq4RUaeGqM63bNni5suWLXPzxYsX12RR3UbnJ9odP2LECDePOj5EHTK8LivR8Uc76WfMmOHmUZ0f\nf/zxbt6snU2aQZnHi+R301mzZo07dv78+W6+aNGiLs6uIuoacfHFF3d5fFRDzz33nJtHx7R9+3Y3\njx4b3vcx+t7W87mYV4YBAACQLRbDAAAAyBaLYQAAAGSLxTAAAACyxWIYAAAA2WrJbhLf//733dzb\nkSxJs2bNcvPoOuGPPvpoTfaDH/zAHfuzn/3Mzb1rgUvSypUr3bysgQNrT+3EiRPdsVHXgEjUZeIj\nH/lIqftpNfXoJrFu3To3f/LJJ9086jIR1ZF3rqNd+bt373bzQYMGubl3TXkp3vE+cuRIN9+5c2dN\nFu1I9jpPSHEngKjLhLfjO3dePZepcSk+b1E9P/jgg27udU2JznH0/DRu3Dg3P+mkk9zcew6VpG3b\ntrm59ziKHs/RfYwZM8bNo049xx3nv57ldQLIqcOEd6xR7ZYV3U/UUWXz5s012cMPP+yO9dYWktTZ\n2enmF154oZtfcsklbt7e3u7mXneT6GfUrl273Dyqxeg5OuoQUeZ5p554ZRgAAADZYjEMAACAbLEY\nBgAAQLZYDAMAACBbx1wMm9mdZrbRzJ6qysaY2TwzW1787V/nEgAAAOjHutJN4i5JX5T0tarsZkkP\npJRuM7Obi8/7TSuBmTNnlsoj5557rptfd911Ndltt93mjl21apWbR90kvF3T3TF48OCaLOomEc1l\n06ZNbn7WWWd1f2KQFF9TfejQoW4edV4YNWqUm0+aNMnNvc4Op5xySqmvOX78eDefPn26m0+ePNnN\no93EHR0dNdm9997rjt26daubR4YPH+7mQ4YMcXNvh3Quu/LLHGfUTcLbSS9Jjz/+uJtHz5fe4+WM\nM85wx1566aVuHj3PjRgxws2j3fR/+tOf3Nx77o66Ruzfv9/No533UQeXqONFmbrNpZ7rJXreiros\nLFq0qCZbsGCBO3bHjh1ufs4557j5q1/9ajd/xSte4ebRc7f3+Ioe01Fnhz179rh5szjmK8MppV9L\nOvonzpsl3V18fLekt9R5XgAAAEDDdfc9wxNSSkcaKK6XNKFO8wEAAAB6TY830KXKa+ZhR2Qzu9HM\nOsysI/q1O9AfUbtoRtQtmhW1i77S3cXwBjObKEnF3/4lrCSllG5PKbWnlNrb2tq6+eWA3kftohlR\nt2hW1C76Sncvx/xDSTdIuq34278WcUaijU9lN5uV3eRXRnSpx2iTyytf+Uo3v+qqq+o2p1YSbUTx\nNid4GxwlacIE/x1Hs2fPdvOxY8e6ebQRw9tYE10uOdpwGeXRXKJNPlu2bHFzbxNVtJkj2ogYbdo7\n88wz3TzaWBfdv6dZNyKV2VgVjY0uRRtdjnj9+vVuHm1OOvvss2uyuXPnumNnzJjh5tFm0+iYog1R\n0WZLb3x0qfNog2t0ienoMVpmAx180fmPajHa5Bhtfp83b15NtmbNGndstF64/PLL3TzaLBptlIs2\nYnrPr7t373bHRptco8d6dBn06PveV7XbldZq35K0QNKZZrbGzP5SlUXwXDNbLunK4nMAAACgqRzz\nleGUUm0fsYor6jwXAAAAoFfxuxQAAABki8UwAAAAssViGAAAANnqbjcJ9HPeTtBrr73WHRvtmv38\n5z/v5tGO59zVo5tEtOM96qYQ7TKPduR65y7qhBJdpjYaH33NaO5RB4Knn366Juvs7HTHRpeMPv/8\n89389NNPd/McLscczTu6vGqZ44wuXbx69Wo33759u5tHXT0uuOCCmiw6l6NHj3bzqPNC9PwXfV+i\nTi1eh4CoU090aeioU0v0nBs95pq1RhupbNeI6Hlrw4YNbv7jH//YzX//+9/XZFHXoAsvvNDN58yZ\n4+YnnXSSm0e1Hh2T9/y6fPlyd+zChQvdPOqcUrYWvZ+XvVHnvDIMAACAbLEYBgAAQLZYDAMAACBb\nLIYBAACQLRbDAAAAyBbdJFrUXXfdVZOtX7/eHTt27Fg3nzp1aj2n1DLqsSs/2h0bXTs+6poQdXaI\n7sfrmuDt3pXiOZa9dvzzzz/v5g8//LCb/+IXv6jJos4TZ555ppu/5jWvcfPx48e7edTdg135LxbV\n+K5du9w86qawd+9eN4+6KZx88sk1WVT70RwPHDjg5lEnjC1btrj5U0895eZPPvlkTbZnzx53bNQ1\n5oQTTnDzqDtA2cdizqK6KNNhQZIeeOABN1+wYIGbe89/USeUyy67zM3b2trcPHp+imrd6zIlSStX\nrqzJvOdhSXr88cdL3Xe0voh+RpV5zq1HJ5wjeCQBAAAgWyyGAQAAkC0WwwAAAMgWi2EAAABki8Uw\nAAAAskU3iSb3zDPPuPmHPvShLt9HtAs2uu45Gifq7BDtnPe6Q0jldp9HO3IjZcevW7fOze+99143\n9zoQRLvvr776ajc/77zz3Dz6PtI1omsOHz7s5lE3iWinfrSTPLJx48aarGznhairSdQ1YMWKFW4e\n7aZfs2ZNTRbV7ejRo908qs9I2cdiLrzvS1SL+/fvd/MlS5a4+bx589z82WefdXOvc0TUTSKql7Ld\nIaJajzqkLFy4sCZbtmyZO3b16tVuHtVi1Amov3VI4ZVhAAAAZIvFMAAAALLFYhgAAADZYjEMAACA\nbB1zMWxmd5rZRjN7qiq7xczWmtni4o+/iwUAAADox7rSTeIuSV+U9LWj8s+llD5d9xmhlB/96Edu\n7u0mfdvb3uaOnT59el3nlKuoI4G3yzYaG+2kLbv7PlKma0I0Nsr37dvn5j/+8Y/dvKOjw829XcZz\n5sxxx771rW9182i3ftStI2dl6jbaMR7tDD/xxBPdfMyYMW4e7Tz3aiXqpBOd46gTRnRMW7dudfOn\nnnrKzb2OGuPGjXPHjh8/3s0HDx7s5n21w75Zeec66rCwfv16N49+tj7xxBNuHtWRd64nTJjgjo06\nW0RdI6JOKNGxep16JGnDhg012Z49e9yx0WM06mwUdciInjP6yjEfYSmlX0vynxUAAACAJtaT/25+\nwMyeKN5G4b8MAwAAAPRj3V0Mf0nSdEmzJa2T9JlooJndaGYdZtaxadOmbn45oPdRu2hG1C2aFbWL\nvtKtxXBKaUNK6VBK6bCkr0i68CXG3p5Sak8ptbe1tXV3nkCvo3bRjKhbNCtqF32lW4thM5tY9em1\nkvwdBQAAAEA/dsztfGb2LUlzJI0zszWSPi5pjpnNlpQkrZL0vgbOEYp3h37ve99zc29n56c+9Sl3\nLDvsG8vbrV+2m0SZHf9lld2pfujQITePdvffc889bh7tnJ45c2ZNdtNNN7ljp06d6ubRrvyyHTLw\nYtH3Keoacd5557l5dH62bNni5l7NRTvso84rQ4cOLZVH9blt2zY390yZMsXNX/ayl7l51AUlOqbo\nuTuXeo6e/7yOB9u3b3fH/uY3v3Hz3/72t26+d+9eN584caKbn3rqqTVZ1GGhbNeIqONJtF7YuHGj\nm+/cubMmi7pGRLV48sknu7l3/JI0bNgwN+8rx1wMp5Suc+I7GjAXAAAAoFfRvBAAAADZYjEMAACA\nbLEYBgAAQLZYDAMAACBb/evi0AjdcYe/Z3H+/Plu/o53vKMmmz59el3nhO4r200i2jVdj24S0VwO\nHz7s5tGO/y9/+ctuvmTJEjePdl/feOONNdkrXvEKd2zUCaBshwzU8upi4ED/R8a4cePc3OtqI0mn\nnHKKm0e75g8cOFCTRbVftmNItIN/5cqVbh51U/E6QVx00UXu2BkzZrj58OHD3Tz6vpfpGtGKHSai\nGti3b19NFp3PhQsXurnXYUGKa93rgiP53RSijgxRB5Ooa8SGDRvcfMeOHW6+evVqN1+zZk2X7zvq\neBJ1jjnnnHPcPKr1vnru5icGAAAAssViGAAAANliMQwAAIBssRgGAABAtthA188sXrzYzT/wgQ+4\neXQZ1E9+8pN1mxPqr+wmn3pslItE971nzx43/+lPf+rm999/v5tHl9597Wtf6+Zvf/vba7Lo0p1s\nlGucMhvoovMQbaCLNuFEm9O8Go3GRpeRjTYnrV27ttT4aOPPhAkTarK5c+e6Y6PjL7tRrhU3xZUR\nbfL1Nr/98Y9/dMdGm83Gjx/v5tHP3GhjnbeZ77nnnnPHRpeMXrFihZtHtbtq1apS470NqtElo9vb\n2938yiuvdPOpU6e6efRzoa9qmp8kAAAAyBaLYQAAAGSLxTAAAACyxWIYAAAA2WIxDAAAgGzRTaKP\n7N27182vu+46N492Tr/zne90cy693FrqtcPW25Uf7Zr/3e9+5+bf/OY33Ty6ZOj555/v5tdff72b\ne7u1c981319EXSOi8zNgwAA3j7omRJ1NvK4B0XNimY4Ukr/bX4ovxztmzBg3v+CCC2qy6HmYrhHl\nlKkLyX9Oi7rjRB1Pom4K0bmLulIsWrSoJos6nqxbt87No04YmzdvdvPoEuPR3L2OD1F3iGuuucbN\nzzzzTDePOqdEzw1erfdG/fPKMAAAALLFYhgAAADZYjEMAACAbLEYBgAAQLZYDAMAACBbx+wmYWan\nSPqapAmSkqTbU0pfMLMxkr4taZqkVZLenlLa1ripNqdot+sb3vAGN1+2bJmbz5w5080/8YlPdG9i\nyJJ3DfpnnnnGHXvfffe5+cKFC928ra3Nza+++mo3nzVrlpt7O55z303f35U9P9H46PmyzH1E+fPP\nP+/mq1evLjUXb+e9JJ177rk12ciRI92xZUXdFHhc+AYNGlSTRc9PUdeIzs5ON4+6UmzYsMHNt22r\nXRZFXSCijjxRd4hI1MEh6uzjdYiYM2eOO3bSpElufvzxx7u5dy6k8p1pGq0rrwwflPThlNLZki6S\n9H4zO1vSzZIeSCnNkPRA8TkAAADQNI65GE4prUspPV583ClpqaTJkt4s6e5i2N2S3tKoSQIAAACN\nUOo9w2Y2TdL5kh6VNCGldKRD9HpV3kbh/ZsbzazDzDo2bdrUg6kCvYvaRTOibtGsqF30lS4vhs1s\nhKT7JN2UUnrRpXlS5Q1N7puaUkq3p5TaU0rt0Xt2gP6I2kUzom7RrKhd9JUuLYbNbJAqC+FvpJTu\nL+INZjaxuH2ipI2NmSIAAADQGF3pJmGS7pC0NKX02aqbfijpBkm3FX//oCEzbHLR7tBf/vKXpe7n\nnnvucfMxY8aUnRIy4HWNkKS1a9fWZPPmzXPH/vznP3fzaBfwq171Kjd/05ve5ObDhw93c3bIt76o\nO0KUl3Ho0CE3j7oDbNmyxc2j+pw4caKbn3XWWTVZtJM+UrZrRC5dJqLjiZ6LvC4eUReQqFPD0KFD\n3Tx6+0bUTcGb4759+9yxkehV8unTp7v5RRdd5OZz58518xkzZtRkJ5xwgjvW6/YjSQMGDHDzsufO\nG98b9XzMxbCkSyX9haQnzWxxkX1MlUXwd8zsLyU9K+ntjZkiAAAA0BjHXAynlB6SFC3Lr6jvdAAA\nAIDewxXoAAAAkC0WwwAAAMgWi2EAAABkqysb6NAFO3bscPNoV2fk61//uptH1xRH3qKd8xs3+p0O\nH3rooZrsV7/6lTs26oTi7TyWpCuu8LcQRDueo93HaH316BoR3cfBgwfdfOfOnW4ejR87dqybT5o0\nyc29DgaRehw/4o4EXmeHKVOmuGOjriHTpk1z8127drl51CFi7969NVnUHWHIkCFuHnWNmjDBvdZZ\nWLvRsXodIsp2h4hE4+vRIaKeXSZ4ZRgAAADZYjEMAACAbLEYBgAAQLZYDAMAACBbLIYBAACQLbpJ\n1MlXv/pVN1+xYkWp+7nsssvcvNWuNY9yDh8+7OadnZ1u/oc//MHNFy5cWJOtWrXKHTt69Gg3b29v\nd/OLL77Yzb2d3VK8ExytI+qaUI88Grt//34393b1S9LgwYPdfNy4cW7e1tZW6n7KKNtlIvefC9Hx\ne+fixBNPdMeOGDHCzaOuIVH3keg52ptjVCuDBg1y86izQ/QcGuVl6qXRHU/KzKU36pyfRgAAAMgW\ni2EAAABki8UwAAAAssViGAAAANliMQwAAIBs0U2iG5YvX16T3XLLLb0/EbScsjvk169f7+ZLly51\n87Vr13Z5LtOmTXPzWbNmuXm0+z73He85KNsdoh6i3ftRPmTIEDc/+eST3TzqGjB27Fg39zoERLVP\n14j6KPMGNBzJAAAFd0lEQVR9iTo1DBzoL4OGDh3arTn1RL3Ocz3uJ7ea45VhAAAAZIvFMAAAALLF\nYhgAAADZYjEMAACAbB1zA52ZnSLpa5ImSEqSbk8pfcHMbpH0V5I2FUM/llL6SaMm2p/Mnz+/Jtu5\nc2ep+5g5c6abDxs2rFtzQnOJNtAcOnTIzaMNdNHlmKPxXn1NnTrVHTtlyhQ3P+2009w82ogSib4H\n3saNem3Eym1TSH9V9jx4l5eNLlEbXeo2urx4dD9RzUWXFx81alRNFm3aqsflcrszPhf97VK/6P+6\n8tProKQPp5QeN7ORkhaa2bzits+llD7duOkBAAAAjXPMxXBKaZ2kdcXHnWa2VNLkRk8MAAAAaLRS\n7xk2s2mSzpf0aBF9wMyeMLM7zcz9HZSZ3WhmHWbWsWnTJm8I0C9Ru2hG1C2aFbWLvtLlxbCZjZB0\nn6SbUko7JX1J0nRJs1V55fgz3r9LKd2eUmpPKbW3tbXVYcpA76B20YyoWzQrahd9pUuLYTMbpMpC\n+BsppfslKaW0IaV0KKV0WNJXJF3YuGkCAAAA9deVbhIm6Q5JS1NKn63KJxbvJ5akayU91ZgpNrdL\nLrnEzefNm+fmdJPIW7SzOerUMGbMGDefPXu2m3s77Xfv3u2OjbpJTJ7sbxmIds6XvVSvl7PLvn/r\ni8vIRmNHjhzp5tFz60knneTmUceHKPceo1Gniug+ItQz0Fhd6SZxqaS/kPSkmS0uso9Jus7MZqvS\nbm2VpPc1ZIYAAABAg3Slm8RDkrz/lmbRUxgAAACtiyvQAQAAIFsshgEAAJAtFsMAAADIVlc20OEo\n73nPe7qUAZFod3i0+3zEiBFuPnz4cDefPn26m19++eU1WdTVIVKvzg716ByA/q2R561spwavk0qj\nUbdAc+CVYQAAAGSLxTAAAACyxWIYAAAA2WIxDAAAgGyxGAYAAEC2rOxO8h59MbNNkp7ttS/YeOMk\nbe7rSfSS/nSsU1NKbb35BandptWfjpO6rY/+dE4bqT8dJ7Xbc/3pfDZafzrWLtVury6GW42ZdaSU\n2vt6Hr0hp2PNQS7nM5fjzEku5zSX48xFTuezGY+Vt0kAAAAgWyyGAQAAkC0Wwz1ze19PoBfldKw5\nyOV85nKcOcnlnOZynLnI6Xw23bHynmEAAABki1eGAQAAkC0Ww11kZnea2UYze6oqG2Nm88xsefH3\n6L6cYz2Y2Slm9qCZPW1mS8zsg0XecseaC2q39Y41B9Rt6x1rLqjd5jtWFsNdd5ek1x2V3SzpgZTS\nDEkPFJ83u4OSPpxSOlvSRZLeb2ZnqzWPNRd3idpttWPNwV2iblvtWHNxl6jdpjpWFsNdlFL6taSt\nR8VvlnR38fHdkt7Sq5NqgJTSupTS48XHnZKWSpqsFjzWXFC7rXesOaBuW+9Yc0HtNt+xshjumQkp\npXXFx+slTejLydSbmU2TdL6kR9Xix5qhlj6f1G7LaulzSd22tJY+n81euyyG6yRV2nK0TGsOMxsh\n6T5JN6WUdlbf1mrHmrtWO5/Ubh5a7VxSt/lotfPZCrXLYrhnNpjZREkq/t7Yx/OpCzMbpEphfyOl\ndH8Rt+SxZqwlzye12/Ja8lxSt1loyfPZKrXLYrhnfijphuLjGyT9oA/nUhdmZpLukLQ0pfTZqpta\n7lgz13Lnk9rNQsudS+o2Gy13PlupdrnoRheZ2bckzZE0TtIGSR+X9H1J35E0RdKzkt6eUjr6TfNN\nxcwukzRf0pOSDhfxx1R5H1BLHWsuqF1qtxlRt9Rts6J2m692WQwDAAAgW7xNAgAAANliMQwAAIBs\nsRgGAABAtlgMAwAAIFsshgEAAJAtFsMAAADIFothAAAAZIvFMAAAALL1/wEufHLDkoXM+QAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11e9ff7f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))\n",
    "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax1.set_title('True image')\n",
    "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax2.set_title('Learned image')\n",
    "ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax3.set_title('Learned image')\n",
    "ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax4.set_title('Learned image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXuUZXV157/7vmjurQbjacQRqFuK4izGqIlE42saLZNg\ngxLHLJUU0s7MsuTmMcSBPLBWBieZMq9ZQSfIYI3iOH1vjMwYHzE4JATGrPggNAhMRHTQrioUH3QB\nAl1IN1V7/rj3FKdOncfvnHve9/tZa6/uuvc8fufcc75nn/3bv/0TVQUhhJDqUMu7AYQQQpKFwk4I\nIRWDwk4IIRWDwk4IIRWDwk4IIRWDwk4IIRWDwk5yRUSWReR1ebeDkCpBYSelQURURJ6b0LbOFpHv\nJLEtQooGhZ0QQioGhZ0UBhF5qYh8WUQeFpHvichVItIafff3o8XuFJHHROSto8/PE5E7Rut8SURe\n6NjesohcJiJ3iciPROQTIrJLRDoAPg/gWaNtPSYiz/Jozz4RuVtEHhWR74rIZaPPzxaR74jIe0Tk\n8Gg/c471zhWRr4rIIyJyn4i817XdV43a+vDo+3eMPj9ORP6ziKyKyA9E5BoROT7Rk0wmAgo7KRIb\nAN4NYA+AlwOYBfArAKCq/3K0zItUdUpVPyEiPwXgWgDvAmAB+BCAz4rIcY5tvgXAOQCeDeCFAN6h\nqkcAvB7A/aNtTanq/R7t+QiAd6nqbgAvAHCT47tnjtp5CoD9AJZE5Pmj744AuAjA0wCcC6AnIr8I\nACLSxfCh8mcATgLwYgB3jNb7QwBnjD577mjb/8Hw3BGyBYWdFAZVvU1Vv6KqT6rqMoZCvTdglXkA\nH1LVW1R1Q1U/BuAJAD/rWOa/qOr9qvoggL/CUDRNOQbgTBE5QVUfUtXbXd//rqo+oapfAPDXGD5E\noKr/R1X/r6puqupdAD7uOI5fBnCjqn5cVY+p6pqq3iEiMjqed6vqg6r6KID3AXhbhPYSAoDCTgqE\niJwhIp8Tke+LyCMYCtuegFW6AC4dhTQeFpGHAZwGwBlW+b7j/+sApiI06c0A9gFYEZEviMjLHd89\nNPL8bVbs/YrIy0TkZhF5QER+BOBix3GcBuBbHvs6CUAbwG2OY/nfo88JiQSFnRSJ/wrgHgDPU9UT\nALwHgAQsfx+ARVV9msPaqvpxg32FljVV1VtV9XwAzwDwaQDXOb7+iVGs3mYagB3O+XMAnwVwmqqe\nCOAax3HcB+B0j90dBvA4gH/hOJYTVTXKg4gQABR2Uix2A3gEwGMi8s8B9Fzf/wDAcxx//zcAF488\nZBGRzqjjcrfBvn4AwBKRE72+FJGWiMyJyImqemzUrk3XYv9xtNyrAZwH4H86juNBVf2xiLwUw/CL\nzQDA60TkLSLSEBFLRF6sqpuj47lSRJ4xasMpIvILBsdCyDYo7KRIXIahCD6Koch9wvX9ewF8bBSq\neIuqHgTwTgBXAXgIwL0A3mGyI1W9B8PY97dH29uRFQPg7QCWR2GhiwHMOb77/mif92Mo1hePtgkM\nO3x/T0QexbDzc8vTV9VVDMM7lwJ4EMOO0xeNvv7t0TF8ZbTPGwHYHbKEGCOcaIOQaIjI2QD6qnpq\n3m0hxAt67IQQUjEo7IQQUjEYiiGEkIpBj50QQipGI4+d7tmzR2dmZvLYNSGElJbbbrvtsKqGDlrL\nRdhnZmZw8ODBPHZNCCGlRURWTJZjKIYQQioGhZ0QQioGhZ0QQioGhZ0QQioGhZ0QQioGhZ0QF4PB\nADMzM6jVapiZmcFgMMi7SYREIpd0R0KKymAwwPz8PNbX1wEAKysrmJ+fBwDMzc0FrUpIYaDHToiD\nhYWFLVG3WV9fx8LCQk4tIiQ6FHZCHKyurkb6nJAiQmEnxMH09HSkz6sK+xnKDYWdEAeLi4tot9vb\nPmu321hcXMypRdlj9zOsrKxAVbf6GSju5YHCToiDubk5LC0todvtQkTQ7XaxtLQ0UR2n7GcoPxR2\nQlzMzc1heXkZm5ubWF5eLrSopxEyWVnxrjPl9zkpHkx3JKSkpJWaWa/XsbGx4fk5KQf02EnpKXtH\nX9z2RwmZRNmHl6gHfU4KiKpmbi95yUuUkCTo9/vabrcVwJa1223t9/uxt9ftdlVEtNvtxt6O6fZ7\nvV7s9ovItvVsE5Ed+3Tvo9VqqWVZnsfZ7XY9t9vtdhM9FyQ6AA6qgcZS2EmpSVKEkn5ImGzfT5xN\n2m967H7L+R1n2ueBxIfCTiYCU6/VhLQ9VROBjdJ+UwH2O0dBx5n2mwuJB4WdlB4TcUlSjJN8SETZ\n/jjtDzpH9ndJPkxIvlDYSakx9UaTDBuEPSScQlmv17e+M92X3/bdgp9E2MPrvCT1MCH5QWEnpcLt\neVqWFSpCtvj1er0toa3X69rr9Yz2EeUhESSUpkLst/1er7ejQ3XcMEiQp25ZljabTd/jZAimuFDY\nSWmI4106RTFJz77f7297qFiWZRTS6HQ6RsfZ6XS21qnVajseQmHtNBXesLCS13bYaVp8KOykNESJ\nA7vN9tS9vFKncDkF1b2cEz9xM2mL35uCvV23lwwM0w6dHrrf8djiayq8cfoekuqvoNefHhR2Uhqi\ndCqmYVNTU1seq5+w+n3uXsaPoIeX6fHXarVA4XUKqmVZ2mq1PB8CfsKbROcxvf50obCTwuEnKH6i\nNzU1ZSSmSYh7o9Hw9KjdAhW2HT8PNe2Hl5egNpvNHYOQvN4c6vV6YJ9GFI897ZTRSYfCTgqF3+Cc\nXq/nOzIyTGiBYWw7K48/rEPX2XZnTNx0vXEs7E3D/jfOuQoKMblJO2V00jEVdhYBI5ngVddEVXHN\nNdcAAI4//vit72u1Go4ePWq03SNHjiTb0ADW1taMljt69CguuugifPGLX8RHPvIR42MZh7D6Lva/\nQ22IxvXXX2+87PT0tGcVyEmbqCRvWASMZILf1HK2uDtFc3NzM6tmpcbm5iauueaaQFG3LAu9Xg+t\nVivDlkUnyrSAnKikGCQm7CJSF5GvisjnktomqQaDwQC1mv+lFseLLANBxyUiOHz4MK6++mrs3r07\nw1ZFJ4q3zYlKikGSoZhLAHwdwAkJbpOUHLtmOEu+bkdVISJ5NyOUVqsV2duem5ujkOdMIh67iJwK\n4FwAH05ie6Q6eMXW42BP8hDk+ZPk2b17N0W6hCR1l7wfwG8B8A2Oisi8iBwUkYMPPPBAQrslRSco\nPtvtdo238/znPx8iUon4e5l48MEH824CicHYwi4i5wH4oareFrScqi6p6lmqetZJJ5007m5JSQiL\nz1qWZbSdu+++u7Kx+LRIYio7ZrOUkyQ89lcCeKOILAP4CwCvFZF+AtslFcArS8JmZWUFjzzySOGz\nQsqIiBj3a9gPAHfMn9ks5WVsYVfVy1X1VFWdAfA2ADep6oVjt4yUnsFgsBVj9/Mejx07ht27d29l\nUZBkiPJ2Mz8/D1XFgQMHtmWz7N+/H5dccglEBCKCPXv2lG4+2YnFZBSTqQE4G8DnwpbjyNP8SLpA\nk9/2olZstOn1epmMIqU9ZV7D/YOKlrHuS36AJQWImzQmfvbbXtSZe5xtSKr+C83c3AT9fqz7kh8U\ndrIDv5olzhs1ikcfVPApak0Suw302LO3Wq224zcP+v1Y9yU/KOwThIkY9/v90Bs16iQPQduLWmPd\nbgO99fyt3W4nVu2RJAso7NXFXXfbb5ozJyav1kEeeL/f31Hf288sy/IUhiDBsCfGyFvUaE/9Hoyx\nFw9Q2MuN3xRtpp2Sbq8q6NXaLssaVHLVtPRss9n0fABYlqW9Xs9zO37r0PIzu9/D6xok+QEKe3np\n9/vaaDQ8BdBUYN1x0HE9dpN91ut138kxvGb0sduZt4hNkrVarW2Tb5j0u5DiAENhZ+GNAnLJJZfg\nySef3PH5sWPHjGuCP/3pT9/2d9BAk5WVFczMzGDfvn2eJVf37dtntM+NjQ089thjnt+tra15lrAd\nXqvED9Pcfnu5sNGm1157LQ4fPozNzU0sLy/jAx/4AMvsVhET9U/a6LEHgwQ8s2azuW1+S5N12u32\ntomVu92u9nq9SPnotOTNfqvxe7up1Wqh85ba5gUnny4PYCgmO5K+MZIUhKixa/crODs0i2F2fNuv\n/yJs/liv35aUDzAUkw12vfGVlRWoKlZWVjA/Pz/W0GvTwlhhqGrkadnc05p5TXOWBZ1OJ5f9FhU7\nBHfttdfuuD7W1ta2rrnFxUU0m03PbZiG1EgFMFH/pK1KHnsas7L7DefOyuwQjF9HaBb7z2IC6LKZ\nSSe3ff10Op0d348zypgUAzAUkw1RZ2V3xrztwTi2kLo/n0SLkvkzaWZfUybXnJ/41+t1xtJLDCjs\n2RDFY49aGGvSzC8lkja0MI/djrVH6SynuJcLUNizIUphLXZExhMs2lNZTvY15xWqazQakUN47FAt\nF6CwZ4c7K8adMmiailZVs9Mow4aoT+r5MbF6vb5tYFFS/R/ukCFTH4sNKOz5EOTBFzl2nFaIyCkO\nvV5vm3hPTU1tewhWsW/Br+ZKUczpsSdd1pkkDyjs+eAXTiiyN2qLb9LbdXrjXqLRarW0Vqvlfvxp\nn9u82+BnbtFOI8OLJAso7PlQZAEPu7nTEjbV8ePntlef9/lK27I6RhHZKv4Wdu2y/npxAAco5UOZ\nZnW3LAtLS0sAgJmZmVT2sbq6uu3fOIgIFhcXS3Vu49BsNnHxxRcbD84aZ45YVcX111+/7TO/85vm\neR8MBpiZmUGtVsPMzAznVE0KE/VP2qrssff7/dLEikVEZ2dnY8fXTbxLO2963HMSpWRx2c3rvLqr\nfdod0uOcD6+O0yxj7IzpRwcMxeRD1aZ2ExE988wzdwhz1gXC7Bu+6J3QaZk9EYnXpOHOz6OcG7+x\nFlllxTCmHx1Q2POhLN66qXU6nR2DhsYR2XE6S22hKXKHZFpmEuf2+z28JjJJ2jOO80CIEtNnGuYQ\nUNjzIW8ByMosy8olLFLmUIxX/RZTC/Ni/cJUzpm3kp4NyfmQdYu0+8HhJcymHjtDNk8BCns+VD19\njxbf4oaQTETMpDBYkuJo0t8Rtm+vUJ7XA8HvLdgvPFVlQGFPD7/XwjJ1nNLKYU7POigcERbWiOId\nm4ilSTjMZN9B+4vaWT4JXjwo7MkR9MrpvKCyjv3mldfdbrcr04FpdwLn3Q4/c86EFeTdhgl30LXi\nvM7d+7DXM32QRN130AMkzv1kWVZ6QlAAQGFPBlOvIesBNPV6XWdnZzMXmnq9rv1+v9BiaGp5PZCT\nNNNQS9CIaHuZsIe1U+SjLGvSt+DlbY9zP1U1NAMKezJEuemzEog8PXWTKdho2ZkzgySogzSsZMQ4\nHbtJmTssNO41Zl+vvV5vK0Rar9efGnF7++2qtZpqibx8ZCXsAE4DcDOAuwF8DcAlYeuUSdijiGin\n0yl0wacwC+ofiPMqTkvfnGIY5LWnUQsoDXM+iJII9/k9sHoXX6z6ileonnwyhd1zA8A/A/DTo//v\nBvBNAGcGrVMmYY/qNRSxsFWYCDvjnL7ezZjnhZaOtVqtbbNveZmdPZJ3W02s0Whk8vZwkYjq6aer\nXn45hd1og8BnAPxc0DJFFXavHvqqDGP388btmHmcc1WF80KrlnWBoax52N7RMlOAfhfQd550kt75\npjdR2EM3BswAWAVwgsd38wAOAjg4PT2d/hmIgN9rnzNGV4XQg58Qx00Tm9Th/WW0SUnDbQH6spHt\nPe44fbmIXgfoOqDPGS3zx4DeOPr/f2o09PGpqRRUJR2QtbADmAJwG4B/FbZskTz2MM+z2+2W5jU2\nyOw3EL8b3KQ+R1D+vv25ZVkU+4Jald6wOp1OqLPV6/X0g69/vW4AetHoszMAPQLoC0Z/XwHoWq2W\nssokB7IUdgBNADcA+PcmyxdJ2MNEW0Qq4a2bTD8XNDAlSo6zyXmlZWtFz9e37Qp4h1F+Ica2ngfo\nQ4B++pRTthyazwP6Qdf+HgAy05txQYadpwLgfwB4v+k6RRL2MNGumscelM/s/Nt0dh2/5csgIpNk\ns7OzRtdx3iGbKzAU45e57ISI25kC9GuA/gOgxzca2u/3df8znqGPAHo6oCeO7A8AfbBWU33oIdUf\n/zgPCYoEMhT2V41O5l0A7hjZvqB1iiTsQRe7M1WsKq+xXtUa/R5uzvCMyVuLPeqvSueLlq1dgaEH\nHXW9CwA9BOjG6N9bAL0f0GeOvu92u3rrhRd6vg1s2e//fk4qZA44QMmMsKp4zuW8vJmXAPpRQO8Z\nXVQfLcDNEWbNZnPbjPd+yzkHv5i+tbBDlRZkXfgL617EE/YLAH3Mta1NQN/rWk7vu0//5j3v0bee\nfLKeDegbTzhB//y44/RhQN968sn6qfe/Pw8JigQo7OaYFD7yE6x/B+j/A/QAhilUHy3AzWN0gzm8\n8bA6I6rmoRWKOi3InFkrtjmzVq4A9BiG4n4U0NsBfVPINg/B+0Fxv2Mfr6jVVH/0o233c7vd3vYg\nKUMRMVDYkyMotCCO/9+K+ML+CkC/AujjgH4b0F9P+QZzeuN+ou0cnFSFfgZa8excYFvWyhyg7wb0\nbEDfAOjnMBTpIHHfgP9bwDa7+eYd1/MV2P6GYJIdliegsCfDp6680vdC2eu6wOIK++mAPgroxwF9\nDaC/jaHX8m9TvKGcxaNMUiCrkBlEK5bZWStXhSz3JQw9d7/vD8H7/jzkWMZdesFvWyYzVeUJKOzj\n0+/39WnHHx/46ui8KOIK+zWAfgPQuuOzDwK6mtINFaVT2MbEY3d3ytJofubMWmmELHsZhl55zed7\nrxj7Y6PP7WXst8+wa96efL2o1SFBYY+OycTA7ldHp/kJe9d10TltL6ArGKZdOdd59ej7F3hsbxwT\nEe31ekaTgtTr9W3nxuuGsAeKmJRypdFs+yS2Z60E2aUIFnZgZ1bMBa7v45RoLmLMHRR2M6L82GGv\njn7CHtRhdCaGAr7ftc6e0ee/lMJNFWW+Uq9z5Rxhaos689ZpfuYW3esAfQLQVxqu/yVAb0ugHfa1\nG3WdIgEKezhR8q1NXh1NQzFOr/9ZGAr4+a5l6qPP35njDel3UVeldg4tffNLRbwJO52d3YDeDOiv\nADoL6C8C+tcY3itvSKg9UQdgFS3mDgp7OH6eutdrncmro4mwu73+Igu710z2/X6fok4ztkPwD0O6\nbS+gHwb0Wxi+zT4G6N8Dek5CbYlz3dJjL5GwB4VfvDyMJzDMUgl7dQwTdi+vvz3ahztmn1YoJup8\npfacmzZMe6RFMb9UxI0CtC3MosTYTScBHxdQ2L0JC78c8rkQH4H3q+MeQN88snsxfMV8M6BXwtzr\nXwH0fa7PXjXabxKdp/brZ9wa851OJ1Z8kkY7BO/76VAB2ua2Vqu1rd8oiqgHzTebJKCwexPmcRoP\ndsDw1XGv4bJBXv81gH4d23v9/wxDwU/qovUSd/tcULBpaZlJKmIW5jerWRLpjSYjt5MCFHZvgkSs\nXq8n4mH4bcPP67cHKA0wHHH3mxgOp05rgJK73G5Z5sOkldPCUhGzsE6n4/uW6tWXlISmpNHxCgq7\nN35PV3uKuMUzzxzbw4jq9QNDT/4WDEsKHEL6JQVss18ZmYNOm2Rz9yUloSn02DMkKFWv2Wxqr9cb\n28M4hPG9/iyNg4totKHFCckwxp6TsGcdTy5KXDGKMc5OK5o1Gg1tNpuZ79dLlMOyXpgVk7Gw5zXp\nQxHiilHMz2PftWtX7m2jldPGdRbsN+g8Umydnaq9Xi8zjzwMUNiHMO96PPPLJqDRgszus+p0OmNt\nx7KsWPdwkp6+yQxjWQFDYW+gwgwGA6ysrOTdjFKzubmZdxNICdnY2MCFF1449nbW1tawtrYWeb1j\nx46NvW+boZ7uZHV1NbF9JE0t7wakxWAwwPz8fN7NqCyWZaFer+fdDEJyY3p6Ou8m+FJZYV9YWMD6\n+nrezagk3W4Xhw8fxq5du/JuCiG50G63sbi4mHczfKmssBf5NaksNBrekbp9+/YBAI4cOZJlcwhJ\njShvn5ZlYWlpCXNzcym2aDxKI+yDwQAzMzOo1WqYmZnBYDAIXL7Ir0lZUavF/3kty8KJJ57o+d11\n110Xev4JKSoisuOzjY0Nz8+9ePzxx7f9HVWbMsGkhzVpi5oVE2cAQF5pjkWzqFktlmXp7Oxs5LrV\nNFqZbNxsL+ecwVmmQqJK6Y4mQ3a9Bgi4Z/zJ+2Iqg3HeUlrVLYnBeHYdmCzLCahWTNjDiuyEPTXp\nvdNotCTNFu4sC4CpVkzYw56KYYW9OEiJRptMc761x/XU3es5nUZ67GMIe5hHHvSD0VOn0SbT3OIa\ndxtBdWD8JnHv9XqRNM4UZCnsAM4B8A0A9wL4nbDl45QUCDq5YR45h8XTaNWzoP4grzK8Ubdv0gla\nWY8dQB3AtwA8B0ALwJ0AzgxaJ+laMf1+P5cqcDQaLT+zLCswKcI9gYZJAoUdsnE7j36OZWVj7ABe\nDuAGx9+XA7g8aJ0khZ0xdBptsi0sFOtMojDJ+vLSGL9QcJU99l8C8GHH328HcFXQOkkJO7NdaDSa\nqdkleIMcwVqttkNngsQ7Sh57EjXbUTRhBzAP4CCAg9PT05EPyAt66skac9hpVTdbdMOWixJuMRHs\npAYyYRJCMZz1J3njOaVV3UxHVScZbkkqZANDYU+iVsytAJ4nIs8WkRaAtwH4bALbDYX1YJJneO0Q\nUl02NjaMlltfX8fCwgL27du3o45M1OqOfkUJUytWaKL+YQZgH4BvYpgdsxC2fJIxdhTAA6DRJsXK\n0KeV9FunO+NORCLnqZfRY4eqXq+qZ6jq6apa3CLFhJCx2L9/f2AVxLwnXxERHDhwAO12O7Ftumdj\nUlVcf/31kbaxuLi4o02p1nQ3Uf+kLSmPnZ2nNFp21ul0Au85u+NwdnY213baBQDT3EecPPVSZcXE\nsSw7T8vw6kijld2cQuc3zD4rM818GcfymMhaNeNQTF6EdZ52u13s378/o9YQMrmo6tYkE0tLS7m2\nxe70tCwrcLl6vY7Z2Vl0u91I2y/6tHgAUGqPPWiAUliqEo1Gq66JiJHXbuLdt1ottSxrrBBKUmAS\nQjGq20sK2PmpJoMLaDRaeaxWq0W6ly3LUlWz+jDdbtd3uVqtlquQu8GkCHsY9NhptGpYlDdwu7qj\nSdkR27t3j7xutVqFEnVVCvsWXj9s3KHznF6PRsvP/GqzBC2vOuzMDRptam/XeX+7K0MWBVDYn8Jv\nPlT3D9npdHx/fLvXP++LO+qNoJp/lgKNloQ5a7OYOllhy7Xbbe31eplOSD0OoLCbY/KKFzYNn22m\nk3o4Lzg7dmhXnwu7GE3ije4LkyGp9OxdgP4doD8E9GFA/wHQnytAu6pm9j2YVFVX5+xIQfsrEqCw\nm2FykZhMjN3pdELnQbTNZDhy0Pr2G4dfSElEtt4+7FfQoLcR2ni2AuiHAD0f0NcB+t8B3QD0DQVo\nW1XM9qyTdFBsgpykonntoLCb4Xeh1Ot13/SmsBFkfuIfpcaEiRfBEEsxzPL47IuA3lSAtpXZnFlu\nXuGSsX+3kBmYgOKFZEBhNyOtqa3GHT7s9XBwPxgYXsnGusDwVvGwvT7rXAXoPQVoe9mt2WzmnrRQ\npJAMKOxmFDm+1uv1djx4nB4Ec/SzsRagL3PZdYCuA/ocn3VuA/SzBWh7auek1drRH+TMJPG6dr2s\nVqsVfoIXd7kE+02iXq9HrvI4LqCwm5HUzCZpEPbQoceej52LYQz9Ip/v/zWG3vzZBWhrUtZoNHxH\nX7rfTp2xcGc4xa+fJywjLW+zw7JTU1Oe32cp7qCwm5NE1bU0MJmSi0XOsrXnAfoQhqEWr+9/GtDH\nAL2yAG1N0vxGYJpcg/ZAnyAPvihvn3HaUa/XM9MEUNjLj59HblnW1oPI2QFke0dFuUmqZlOAfg3D\ndMaGx/fPBvR7gH4G0FoB2pu02cP0Ta5Rt3U6Hd9BQqZT1aVtduZNnJh+VmASqjtWHa/i/K1WC488\n8ghWVlagqlhbW8Pjjz+Ofr+PJ598EqqKAwcOoNFo5NTq6nABgEMANkb//h2An8Bw9vYnXcueBOAG\nACsYzg25mV0zM2NtbW3HZ6ZTux05csR3SjrTqeqSwLIsdLtdiAgsy4JlWRARdLtdLC0t4eqrr8bU\n1FSkbeY9uYgnJuqftNFjN8cdJvLzJpydvVFG5tG87QIMQyrqsE1A3+uxbAfQWwH9JqB7CtD2NM1N\nmfp5gvrOnPdZnG1nFcIFQzHVJOjCq9VqOjs7y7h7AnYI20XdtvuxPTtmN6A3APoEhg8Dd/ZM3seR\npDlDMXmWxHaP1Lbb4b436vX61kjwoAyWKH1VjUbD9x7MIukCFPbq4PQmihKPrLptwFvY3bY35Ht7\nez0MvfoHAT0C6F2jz/I+TlNzVjrMu9PeT0Cd94llWTsmofZbz/QBVavVth4OeaVJg8JeDfK+iSbV\nDsFbqA/F3N57RnYeoK8FdBHDh8elBTjWMJuamopUd8jZue83anTcDJkwAY0ivFHCL/bDIa2BjWGA\nwl58TEoT0EPPx7xi7I+NPk9qH31A7yzAsYaZPeLZNAbtJW5eue5ujxoYvhn45YuH7cNJ0LpuooaU\nut0uPXYvo7CHD4yip56/XYChh74x+jdI1LvwD8fs9VnnA4B+vQDHmbSZiptfDfRxPfagqe68cs6j\n3mv2xBx5DGwEhb3YBD3x6amXz0zLDtQxzKI5B8OBTr9WgLYnbe5JKqIOAAzzoG1hjbu+F+74fFBW\nmf1QyWNgIyjsxSbIK/F6RaWVy7zKDpyM7Z787xWgnWmaZVm+8XVnhopXmMbPgzapkBp0b0UJlRSx\n3Ago7MXGz6vgqNHym1/ZgTqgL8EwNHMFoI8D+psFaG+aFtTJ6Dcnqbv2utck9W5MMsfCPP2w7Rah\n3Ago7MXG74LO+0akjWdhZQec9rsYhmqOL0C78zA7e8bruzDP2h06CasQKSI6OztbKJGOAyjsxcOv\nCp7zQsv7ZqONZ5/EcBDTMw2WPRfDkMzpBWh30Swo68W0s9M5WY7fvKZe92CRQRbCDuBPANwD4C4A\nnwLwNJN/GYuVAAAQmklEQVT1JlHYTeN1LAVQHnNnzVyH4QjUVxqu/z4MwzGT6rEHWZDHbpqe6Hw4\nmIY+846hh4GMhP3nATRG//8jAH9kst4kCrvpKye99nKYXy2Zm7AzO2Y3oP8I6K9iOCfq6wH9U0CP\nAvoHBTiWollYB6lpP5Tz3oqy/7Rz0ccBWYdiALwJwMBk2UkU9rCeeuerIOcyLb4dwnZRD7K9gC4B\n+g0Mywk8AOiXAJ0rwHEU1bzEPUp9GveYkChJCWmPHh0H5CDsfwXgwoDv5wEcBHBweno69ROQBVF6\nzKNekEWeUYbmX0tmowBtq4o5s1jC4urOuVHdk2BHHRMyER47gBsB/JOHne9YZgHDGLuY7LQKHnvU\nHNd+v79Vac7kgs77pqIF2yF4C/uhArStSmZXlAxyjGynKs5o7aLlqYeBrDx2AO8A8GUAbdN1qiDs\nUdO0GDuvlmVRS4Y2tKCEApMOUj+zHwjMitkp6ucAuBvASVHWq4KwR63uFvWioxXfotSSmUQzfUMd\nx5yOVJwqjUEUUfSRkbDfC+A+AHeM7BqT9aog7FE9doZXaEW2tK7PNNN33eIcxXnq9XqBwl3EcgKq\nGYZi4lgVhN3kh+cUdbSimt3Z6BzAk8Z+ugHTObotyr3i5UFHyX5pNps7Rqs679+4I2LTBhT29Al7\n4rOYF62I5iWKaYYKe71e6JB/u00mnZ9B4jruA8redl4TaYQBCnu+MKZOK7o5y+uaerpx4+adTmfL\nIw8a7el0lrxSfk3CIb1eL3ZoyRZueuwUdk8YU6eVwZrNpvHAn3a7rbOzs7H3ZYuySaeklziblOy1\niTtPsC3cjLFT2D2hx04ri9nC5xbSVqu1FYe3LMtoyrowM/F4g2LlcTxmL5EOi7Hb601kVkxcmwRh\nZ4ydVkazBdUpZFEG/oSFakxi1EFOUdwYt5dIF1G4wwCFPX+YFUPL26KGIoCdXnGUa9iyrMCHgInH\nHRTGdJbiLYMQJw0o7MWCk1PT8jD72ov69ui8bqOsZ9d38XoY+HWSukU6TqGvSQEU9uIRpTodjZaE\nOa89U8+7Xq9vrRdnmL77eneLd1jHZBQnKO8slawBhb24MGOGlpXFzVePc62aetAmqYTuh4LfPvPO\nK88aUNiLC+PutKzMK9MjzBt2CqzpgyBKzDvO4J+i5pVnDSjsxYXCTsvS7NK3NmEhQefApbAYe5w4\ndxyRLmpeedaAwl5cyh6KyaJqHy1Z86qrEua52wLvN+mLcyKMKHjt274nLMvaVsMmSl55GdMXowIK\ne3EpcweqPfqPOfrlsnq9rr1eb0v4TB/O7Xbbd2CS+00gCs63hiBHx9QrnxSPHhT24tLv90OLIjlt\ndna2MA+DXq/HUBJNgWQ6Lk2ua5M4+qTE4GEo7DWQzJmbm8Pu3buNl//yl7+M5z73uSm2KJxOp4N+\nv4+rr74aa2trubaFFIPp6emxt7G6uprqMibrVhEKe8YMBgPMzMxEEsf19XXcdNNNKbYqnCNHjuDC\nCy9Eo9HItR2kGLTbbSwuLgJ46pqu1WqYmZnBYDAw3o7Jw2GcZZJ4+JQSE7c+aZvUUAxHn5bf7Jht\n0DKNRiP3dqZpUQYbjXtPMMa+HTDGXjzC4ontdjvx+PXs7OxWrRBms4xvNkH1VyzL8u1cnpqaKnwf\nRVD73DHrJGLbzmyWoKyYKNthVgyFPTOCev+7o+nJkr7p3Z20ZU+1zNtsxtnGrl27cj+OIPMTa6/0\nxqLONFRVkQeFvXgEeTdlDdOIyEQ9LMJ+yyKa14QVfsva3rLf926xLGI2SpXDMqCwF4+gCy5roWBY\nJro5xcpvbs1Go1HIUItXLXL321yr1Yp0Lbbbbe31eoUT0SI+bJICFPZi4veKOElebxnNLVZ+4mGP\n1owyTiFtswXNfe05ByyFdYgGbbtoYY+ihoeSABT2clGmV/tJM/vtxunt+i1rx6HzGJnbaDR27Nc5\nz2iYZ93r9bZ1tPuVEnAfb9Ggx66gsBeEuDH2Xbt2RZ4hp8yWtyfcarUCRbvb7eb2kG42mzu8cPvv\noPaq+oeWwqyIYskYu4LCXiA4EUe5zRaPPNvgFFoTZ8H2uOM4B0UWy6KFh5ICFPbywnj75JplWWO9\nldhC3e/3jcTafhDEaWdVxLJMgLViyovfMGgRybgl43Pccceh3W4nuk3LshLfZt602230+30cPnwY\n1157LSzLirWd6elpDAYDzM/PY2NjI3SfdlmAer1uvA/LsnD48GHMzc3FaiPJABP1DzMAl2L4JN9j\nsjw99mD8YoRx46B5W6/XM+qIM7FWqxUpnbBIw/ud4YFOp7PVKWuX1HUTNW3SvkZMPXV3x6nJ+Sty\n+GUSQFahGACnAbgBwAoo7InhFyMsYwy+2+0m1sEbNdtkdnY29+O3z4HztzXp3PNarlarbT0Q7MwV\nZ0dpWEw9SJidDwX7YVPVWHVZyVLY/xeAFwFYprCnTxlHqCbVZxDn4ZDkQKywUZl+1mw2t9U/8fPE\nvTJM3HVU3PF3p1CHPfTr9TqFueRkIuwAzgfwgdH/A4UdwDyAgwAOTk9PZ3AKqkvYzV40S8JjzzKk\n4jeaMs6I0k6nY/z7hOWEh+VnJzETESk2iQk7gBsB/JOHnQ/gFgAnqoGwO40eezBRX3/dQl+kaets\nQQnqH7AsS2dnZ32FaWpqKrNjsgcYef0GUb31qOmrYTnhYSMq/fZFT706JCbsvisCPwnghyNBXwbw\nJIBVAM8MW5fC7k8SgytMU93SMHsaP6+HklcM1/7cS7Q6nU5udXS84t1RzqktpqYPA5PfOMxjr/LA\nHDIkdWHfsSF67ImQ1HDoOB6m2+yJq3u9nnGsul6vh9YiidLOvPoTnIIYt1/DJHzjdW783thMhJud\nndWGwl5S4hQw8ruZ/TzhKGYLdBJxfDsdz9nWtCshhpUACDt21fEykYI6XP06S/1SXe122G8OFO7J\nI3Nhj2IUdn9MPPYomRLOUEbcDJEkR8KmPaq2Xq/r1NTUNmGNm0dvP0zHabP91uPehl+IxO/3N12f\nVBsKe0kJe902DQv4hW6KWCs8bWs2m7HeOJLw2O3QlD3lm/2ZvX3TGYmi/MakulDYS0xQnNRUZPxC\nN3mLbFnM/TBN4k3DKyxkWuc9ym9MqoupsLNWTAGZm5vD8vIyNjc3sby8vK0mx+rqqtE2/OrNhJFH\nPRrLsmLXRhmXer2OXq+Hfr+PbrcLEUG328XS0tK2855EbZqjR4/i2LFj2z5bX1/H/v37MRgMAACL\ni4s79uX3m8T9jckEYKL+SRs99viYeHRB8dewGei9BuckGRcP6w9Iaj9xzF2xMEo20LjmPg/O32lq\nampHO5rN5o5+FGbCVB8wFFNNvGLsziHrdizX7yYPmuvSuYxTKExqkEQRTz8RCppSrl6vez4Ukiou\n5hbMKAXXknrwBeWju83+zZi7PllQ2CvMOHnOQesHkUTqpC2CfoS9jXg9FNKondM1LIHgbEdSndIm\n58HZzqBl6b1XDwr7BJLU4KYo247rlXoR9uDweygk9dBx7ifqcSQ1gXWUYxGR0GXpvVcLCvsEkubs\n7EkIZ9jbQ9g+/HL5owz1N91P2Da96qcnVcrBmYcf1k6TBy7TIqsDhX0CKaLHHhTyiRpGsTs30yxd\nbNeJCYux+53TpNoW5v3bD8ko85qS8kNhn0DS7EiLI1hhD5Q4DwsRMfZo45hlWVvtCxJ30xIPcT14\nd39CUN2dsEJp9NirA4V9Qkkz9W3cTls3RZy02y3YYemhJiWV46SPxvGymSFTfSjsJFPiPFCS6pC1\nzV1Z0inKziH9Qeb2bsPW8RNOdz0frxTUoDeCuF42c9qrDYWdpIapeIQtFxbeierRuwcYebUnaH9e\nIh03Q8bLS/fqcI1SIIwQCjtJhSi58lFz6t2ebVDqn58HHiaKJp60E5O3CnfYxG8de3amoDbRyyZB\nUNhJKphm3iSVoRPk0aaZBWRj0mns3l+Ql8+OTDIOFHaSCqa58knm1Pt5tEnn7Qd1DtsPEZOwSZCX\nz9RDMg4UdpIKWXvsSbTFhCTLMQSV+aXHTsaBwk5SIekYexZtMSHpBxE7RUkaUNhJaiSVFZNlW8JI\noxwDO0VJ0pgKuwyXzZazzjpLDx48mPl+STUYDAZYWFjA6uoqpqensbi4uG1SjDjMzMxgZWVlx+fd\nbhfLy8tjbZuQpBCR21T1rLDlOIMSKRWDwQDz8/NYWVmBqmJlZQXz8/NbMxDFxWvmona7jcXFxbG2\nS0geUNhJqVhYWMD6+vq2z9bX17GwsABgKPwzMzOo1WqYmZkxFvy5uTksLS0FTo9HSFlgKIaUilqt\nBq9rVkRw4MABzM/PbxP+drtNgSaVgaEYUkn8JnCenp4O9eYJmRQo7KRUBMXCV1dXPdfx+5yQqkJh\nJ6UiKBYe5M0TMklQ2EnpmJubw/LyMjY3N7G8vLwVP2dmCyFDxhZ2Efl1EblHRL4mIn+cRKMIiQMz\nWwgZMlZWjIi8BsACgHNV9QkReYaq/jBsPWbFEEJIdLLKiukB+ENVfQIATESdEEJIuowr7GcAeLWI\n3CIiXxCRn/FbUETmReSgiBx84IEHxtwtIYQQPxphC4jIjQCe6fHVwmj9pwP4WQA/A+A6EXmOesR3\nVHUJwBIwDMWM02hCCCH+hAq7qr7O7zsR6QH4y5GQ/6OIbALYA4AuOSGE5MS4oZhPA3gNAIjIGQBa\nAA6P2yhCCCHxGTcrpgXgWgAvBnAUwGWqepPBeg8A2FkjNX32YHIfPDz2yYTHXi26qnpS2EK5FAHL\nCxE5aJIqVEV47Dz2SWOSj50jTwkhpGJQ2AkhpGJMmrAv5d2AHOGxTyY89glkomLshBAyCUyax04I\nIZWHwk4IIRVjYoVdRC4VERWRPXm3JStE5E9GJZbvEpFPicjT8m5T2ojIOSLyDRG5V0R+J+/2ZIWI\nnCYiN4vI3aOS2pfk3aasEZG6iHxVRD6Xd1uyZiKFXUROA/DzACZtzrS/BfACVX0hgG8CuDzn9qSK\niNQBfBDA6wGcCeACETkz31ZlxpMALlXVMzGs5fSrE3TsNpcA+HrejciDiRR2AFcC+C0AE9VzrKp/\no6pPjv78CoBT82xPBrwUwL2q+m1VPQrgLwCcn3ObMkFVv6eqt4/+/yiGAndKvq3KDhE5FcC5AD6c\nd1vyYOKEXUTOB/BdVb0z77bkzL8B8Pm8G5EypwC4z/H3dzBB4mYjIjMAfgrALfm2JFPej6Hztpl3\nQ/IgtLpjGQkpNfweDMMwlSTo2FX1M6NlFjB8VR9k2TaSPSIyBeCTAH5DVR/Juz1ZICLnAfihqt4m\nImfn3Z48qKSw+5UaFpGfBPBsAHeKCDAMRdwuIi9V1e9n2MTUCCqzDAAi8g4A5wGY9aqbXzG+C+A0\nx9+njj6bCESkiaGoD1T1L/NuT4a8EsAbRWQfgF0AThCRvqpemHO7MmOiByiJyDKAs1S1ahXgPBGR\ncwD8KYC9qlr5mvki0sCwk3gWQ0G/FcAvq+rXcm1YBsjQc/kYgAdV9Tfybk9ejDz2y1T1vLzbkiUT\nF2OfcK4CsBvA34rIHSJyTd4NSpNRR/GvAbgBw87D6yZB1Ee8EsDbAbx29FvfMfJgyQQw0R47IYRU\nEXrshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMf4/1HOo5p+uW1YAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12fab6eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAAC3CAYAAAD3oFO8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmQXeV55/HfA4jFyAIJLQgEEmKRWAIiFhgHs4Ultkmw\nQ7m8ToaUE5OZSjLOjCsVT6pmxjVVmXFSWWpqqiYTMvaAPTF2cELhcpwhBisDGErQCAm0IIRAWAK0\ngQFhK2x6549uOfee99fdr+7St889309Vl3Qenb73vec55/Sr2+9zn0gpCQAAAGiiwwY9AAAAAGBQ\nmAwDAACgsZgMAwAAoLGYDAMAAKCxmAwDAACgsZgMAwAAoLGYDAMAAKCxmAwXilF/GBEvjX39YUTE\noMeFQxcRV0XEqoh4NSK2DXo86FxE/G5ErI+IfRHxbET87qDHhM5ExL+NiGci4rWIeCEi/iwijhj0\nuNCZiDgyIjZFxI5BjwWdiYgvRsRbEfF6y9fSQY+rH5gMl7tZ0kckXSDpfEm/JOk3BjoidOrHkr4i\niYlT/YWkfylptqQPSPqtiPjEYIeEDn1b0kUppVmSztPovfbfDHZI6MLvStoz6EGga99MKc1s+Xpm\n0APqBybDFRHx8cr/gt6IiH+UdJOkP0kp7UgpPS/pjyX96iDHiomNl8uU0sMppa9JGsqLehhNkMs/\nSimtSSm9nVLaLOkuSZcOerwY3wS53JpSeungbpIOSDpjgEPFBCb4WamIOE3Sv5D0Xwc6SBSZKJdN\nwWS4IqX00/8FSTpJoxOm2yWdK2ldy67rxmKYpibIJWqmJJdjy5Yuk7RhAENEoYlyGRGfiojXJO3V\n6DvDfzG4kWIik1yT/13S70vaP6jxodwkufyliHg5IjZExL8e3Cj7i8nwOCLiMElfl/SPKaW/kDRT\n0qstu7wmaSbrhqc/k0vU1CS5/KJG72n/e6rHhUPncplS+vrYMomzJP1PSbsGOEQUqOYxIn5Z0uEp\npTsHPDQcInNN/rWksyXNk/RZSf8xIj45wCH2DZPh8f2BpHfrn9esvS5pVsu/Hyfp9ZRSmuqB4ZBV\nc4n6srmMiN/S6Nrh61NKbwxiYDhk416XKaUtGn2H/39M9aBwyH6ax4g4VtIfiXttXbVdkymljSml\nF1JK76SUHpT03yR9dJAD7BcqdY2xApxParSY462x8AaN/tru4bHtC8SvY6e9cXKJGhovlxHxGUlf\nkHR5SonK9RoovC6PkHT61I0Kh6qax4g4V9ISSfeP/dL0SEnHRcROSZeklLYNaqyYWOE1mTS6nn/o\n8M5wRURcqNH1Th9JKbVWwn5V0r+LiJMj4mRJn5d06wCGiELj5TIiDouIoyXNGN2MoyPiyEGNE5Ob\nIJeflvRfJF07rFXOw2aCXP56RMwf+/s5kv69pHsHM0pMZpw8rpd0iqQVY1+/rtGlLiskbR/EODG5\nCa7JD0fE7LGPlr1Y0uc0WqQ8dHhnOPdhjX5M0wMty4Hvl/QhSUslPTEW+1+iuGO6Gy+XfyhpVct+\n+yX9P0lXTuXgcEjGy+VySSdIeqQl/n9SSv9qykeIUuPlcqekP4iImRr9SK47JP2HgYwQJWweU0of\nPLgRES9LOpBS2jmA8aHceNfkKxr9GNKjJO2Q9KWU0m0DGWGfBUteAQAA0FQskwAAAEBjMRkGAABA\nYzEZBgAAQGMxGQYAAEBjdTUZjogPRMTmiHg6Ir7Qq0EBAAAAU6HjT5OIiMMlPSXpWo1+5MYjkj6Z\nUto43vfMnTs3LVmypKPnQ/ceffTRvSmleb14LHI5ONu2bdPevXt79sHn5HJweplL8jhY3F+HB7kc\nHqW57OZzhi+W9PTBD7qPiG9o9LPqxp0ML1myRCMjI108JboREc/16rHI5eCsXLmyp49HLgenl7kk\nj4PF/XV4kMvhUZrLbpZJnKz2jjI7xmLVgdwcESMRMbJnz57qP6NGyOXwIJfDgTwOD3I5PMhl/fS9\ngC6ldEtKaWVKaeW8eT35rQMGhFwOD3I5HMjj8CCXw4Nc1k83k+HnNdqD/KBFYzEAAACgFrqZDD8i\n6cyIOC0ijpT0CUnf7s2wAAAAgP7ruIAupfR2RPyWpLslHS7pKymlDT0bGQAAANBn3XyahFJK35X0\n3R6NBQAAAJhSdKADAABAYzEZBgAAQGN1tUwCAA5VaddLt5+LRZQ1cHP7lX4vyvJWmrNSpTkjj4em\nm5x0+vi9PDd6fV7U+fypUy6n4l7daS55ZxgAAACNxWQYAAAAjcVkGAAAAI3FZBgAAACNRQHdgPS6\nqASYrjotvDpw4EDRfu56OOww/p/fjV4Wy7k8Ok0oduq3bopTe/mcLuel13OJ0gIrdx+oy71hEIVx\nTje5rMa6KXx0eevlvaAeZwUAAADQB0yGAQAA0FhMhgEAANBYjVsz3M2aqrfffjuLvf7661lsx44d\nWWzfvn1t2yeddFK2z/z587PYUUcdlcX6vXamLqYil9u3b89ir776atv2okWLsn0WLFiQxYYtl51+\nGHuv1weXHq9OPyi+9DyrQzOIXjY8GVSDDad0XXKnBp3H6bIW2MXeeeedLPbmm29msVdeeSWL7dmz\nJ4u99NJLk47tXe96VxZz99y5c+cWfe8RR0zdVGi65NJdM2+99VYW+/GPf5zFXI5crDrvOfroo7N9\njjvuuCw2b968ov3c4x1++OFZrATvDAMAAKCxmAwDAACgsZgMAwAAoLG6WigTEdsk7ZP0jqS3U0or\nezEoAAAAYCr0YtX4VSmlvT14nIEpLRRwRVc//OEPs9gdd9yRxaqFAldeeWW2zzXXXJPFZsyYkcUG\nXcwxDFzxwAsvvJDFvvWtb2WxXbt2tW1fffXV2T4ul0ceeWTR2EoLx/qlm2YKLlYtsHHXkYu5whx3\nPbiYK0wsKf5x4y/9AHhXuFG6Xz90WuAoleWxmwYbpcfAHfuS886dO6XFqqWNGvp9TU5WQFV6n+hl\nAxV3nf7TP/1TFnNF5A8++GAWe/TRR7PYzp0727bfeOONbJ/jjz8+i5111llZ7Kqrrspi559/fhZz\nxVm91Ekue/lcpUWOe/fmU7knnngii/3gBz/IYlu3bs1i1aI6V6i4cOHCLHbBBRdkscsuuyyLuZzP\nnDkzi5VgmQQAAAAaq9vJcJJ0T0Q8GhE392JAAAAAwFTpdjL8/pTSCkkflPSbEXF5dYeIuDkiRiJi\nxH2mIOqDXA4PcjkcyOPwIJfDg1zWT1eT4ZTS82N/7pZ0p6SLzT63pJRWppRWug9SRn2Qy+FBLocD\neRwe5HJ4kMv66biALiKOlXRYSmnf2N+vk/SfezayHuhmUbr7XrfgfNOmTVlsw4YNkz6+6/TiijRw\naEoLQdzxf/rpp7PY+vXrJ3089/iuQGg6Fz62voaS4inJF9O4a6TasW/37t3ZPtWixPEe/9hjj81i\nxxxzTBZzRXUuJ9WCDvd97jldzHVDcgUjredBLwtnqo9XmkcXc9fHT37yk7Zt15mquk91TAe5Y+UK\nTCc7fhM9R5XrAOli7hwovcanUun9pOR4lf68c13GNm7cmMXuv//+LLZ69eos9swzz2Sx6nnlzkV3\nrmzevDmLufG6n7MXXXRRFuuXXhY+uv3c9bx///4s9uyzz2axe+65J4s9/PDDWcwV1b388stZrHo/\ncK/JFby5YjzXwfDGG2/MYsuWLctiJbr5NIkFku4cS+IRkr6eUvq/XTweAAAAMKU6ngynlJ6RlH/+\nBQAAAFAT/F4eAAAAjcVkGAAAAI3Viw50A9Hvji2u+MQtQnfFcq4r3amnnjrhtuSLOaZz0VWv9LqI\nqOTxXVHG448/nsVckcHixYvbthctWpTt4wo8uil46bfWY1RaLOc6Q/3oRz/KYtu2bWvbdsfZffyQ\nK55ynafe/e53F32vK4yq7ucqv+fPn5/F3LXqTHUuW+9b7hx3eXT3NZfH7du3t2274qfXXnsti7lj\nNWfOnCw2e/bsou8tKTR2RZXuOd35VNpJsN/3rV4p+fnmzgGX33/4h3/IYiMjI1nMFUC5c8rda6rF\nlS4f7t5TPT8l6aGHHspis2bNymLVe3o/dXNPKOku5wpb165dm8XuvPPOLOaK5aodASV/b3H312pn\nP/d9roPhli1bsti73vWuLLZgwYIs5jraleCdYQAAADQWk2EAAAA0FpNhAAAANBaTYQAAADRWLQro\n+l0s57gCOtdh5cknn8xirlPKe97znrZtV3TlFqC7wo06F9UNoujEPafL0WOPPZbF9u7dm8VWrFjR\ntn3yySdn+5R2zpouuWw9RqUFpa6IxR2vp556qm3bFZi689wVsy1dujSLuQI6V5jjCniqxSbuGncF\nN6Xn8VTmPKXUlid3DFyxiuvS5e5r69ata9t2XQNdkYu717kOdO7+516Dy1E1j66A1Z3Dbhwu5gz6\n2u3mZ1m1YM51kbv99tuz2AMPPJDFSq8ZVzReLbBy3+uKPqv3FMkX0D3//PNZzN3nXSfZfum0s5xU\nlss1a9Zk+9x2221ZzOXSFd+5ItPTTz89i7n7dbW7nOtQ6c49lzeX8+o9SZIuvfTSLFaCd4YBAADQ\nWEyGAQAA0FhMhgEAANBYTIYBAADQWLUooOs3t1DdLdp3HXncwnu3yP2ss85q2z7hhBOyfVynHRya\nkg49Ut4VTfKL8V3HnGoRl+ucVdIlazppLQYqLfpyx8YVY1WLIdy1ddppp2Wxiy++OIudeOKJWcwV\nXrlCDZeTasHfm2++me1z0kknZTGntNh1smPdjckKId3rc4Vw7lp4+umn27ZdkZorrKkWnErS3Llz\ns5jLoyvocbndvXt327Z7na64z3UX7KY4sl/cmEoLrFzRZLVzqiuwuvvuu7OYKzx2hVPnnXdeFnPX\nuLu2qo/nuhq682fVqlVZbMeOHVmseh5L/nyfSt3k8oknnmjb/upXv5rtc88992Sxffv2ZTHXve2S\nSy7JYmeffXYWW7JkSRarFki6TqPu5+d9992Xxdz3ug6xzz33XBYrUa+f2AAAAEAPMRkGAABAY006\nGY6Ir0TE7ohY3xKbExHfi4gtY3/m73MDAAAA01zJO8O3SvpAJfYFSfemlM6UdO/YNgAAAFArkxbQ\npZTui4gllfCHJV059vfbJP2jpN/r4bjalBYqdNrdrLTQZPXq1Vls586dWcwVBVSLgVyXo+ncoaxX\nOn093XSucwVb3//+97NYtQhH8oWO73//+9u2S3NZF6XFf66YwxWZvvjii23b1a5Eku8s564j17HK\nHWtXNOmKsV544YUsVuUKf5zS67ef58ZkxXnuvuYKTlw3r2rBpDsuF154YRZbvHhxFjv22GOzmFN6\nLlavXVfo5c6dZcuWZbFBdMmcTDddITdv3pzFvva1r7Vtu2K50vvhFVdckcV+4Rd+IYudccYZWcx1\nj6x273QFdO68cMVUroDu1VdfzWKdFl31isulu1a3bt2axb7+9a+3bd97773ZPq4zqLu/XnvttVns\n53/+57NY9QMBJN91sPqhAK4LqPs54rqUuvPRFf+67y3R6ZrhBSmlgz/hdkpa0OHjAAAAAAPTdQFd\nGv0v67j/lY6ImyNiJCJG3EdjoD7I5fAgl8OhNY/u3R/UB9fk8CCX9dPpZHhXRCyUpLE/8/evx6SU\nbkkprUwprXSfR4j6IJfDg1wOh9Y8us9eRX1wTQ4Pclk/nTbd+LakmyR9aezPu3o2ogFw63Vc84CR\nkZEs5ta7uLVR1Q+krq6LksrXEtZ5PWq/ubV1bs3Z/fffn8XcejuXy+p6KdcspddrRfud88ke310j\nL7/8chZza4ar67rch6y7tYjHHHNMFnPXTen64C1btmSx6gfwu8YM7rGc6bDmf7Ln279/fxZz66Zf\nf/31LFZdG++ujUWLFmUxt1bXNdhweXTXs1t3WD3H3Ll5yimnZDHXOGY6Km0M5dbIfvOb38xi1ZoJ\n91uF448/Pov94i/+Yhb7+Mc/nsWWL1+exdy15c7X6r3GNXdxuZwzZ04Wc2uL3X3erTHvl9LGUK4e\n6Y477shi1fXe7p1o9x/lG264IYt95CMfyWJufbBb6+1+DrrXVeXWLpfm0jUOcWvCS5R8tNrtkh6S\ntCwidkTEr2l0EnxtRGyRdM3YNgAAAFArJZ8m8clx/unqHo8FAAAAmFJ0oAMAAEBjMRkGAABAY3Va\nQFdbpYvXq4U1krRu3bos5hZ1X3DBBVmsuiC810VXFNWNcoVeroHAhg0bspgrzrrsssuyWLUAzJ0D\nw5Y3d424D0F3H3heLQQ555xzsn1ckYzjCp5cgaRrNLB+/fos9uSTT7Ztn3nmmdk+JUUg05G717nC\nOFds5ooGq+e9K2JyhTWuWM5xTQZcMYwrEqs2TXCFze7x3f1iOiot8r7zzjuz2KpVq7JY9Zp0jXCu\nv/76LPbpT386i5177rlZzBW/lhTLudhRRx2V7eOatrgCPXdPd/cQd270i3vNrhDMNUJxsWpTI3cN\nulx+9KMfzWJnn312FnPnhsulu99U93P3eff4bj/3c9Ydy07v17wzDAAAgMZiMgwAAIDGYjIMAACA\nxmIyDAAAgMZqXAGd4xbPP/jgg1nMdelZuHBhFrv88suzWLUIwC0Gd+pSYDUIpV2Z7rvvvizmOg4t\nWLAgi1177bVZrLq4f9hy6Y6rK0iqFm5IvptZtcDNFWe5a8sVUbhrdevWrVnsoYceymJr1qzJYtWC\nP3c9u4Ibd565wg1XwDNVXHGJKzZ0RWqu0K76eO71uvy4fLv9XOesTZs2ZTGX72rMFVOV5tHF3LF0\n10kvtT7+T37yk+zfq13kJOnv/u7vsti2bduyWPW8dIXCn/jEJ7KYK351x9rdE93xKrl3lhYou3PK\nnaOuoNO9hl5qfe1unO48//u///ss5s79ai6vuOKKbB/XJdAVPrrCRFfs75RcI6U/A93PG3ddukJN\n9xpK8M4wAAAAGovJMAAAABqLyTAAAAAai8kwAAAAGqtxBXRuEb/rwPTAAw9kMbeA23Wbc12sSgpp\n6lJgNZ25Tj733ntvFnOFFZdcckkWO+OMM7JYSUFBnXPprhFX9OGKetx+1WKsp556KtvHdZlyRS0u\nv65ob8uWLVnMdZXcv39/27YrMPvRj36UxVxRWGnxSeu50c8iLHeOu3G7/VwBS7XjoCtIdI/vjoE7\npi6PrqPhE088kcWqxXcnnHBCto87X9143X6lHbF6JaXU9vPGnc/f+c53spjrrOkKB6udxq677rpJ\n95Gko48+Oot1022zpGuZK8xy548r6HWP7zq0nXrqqVmsV1JKba/BdQ50hY+PPPJIFnPX5fLly9u2\nP/jBD2b7uGI5d38t7Yxbkjcpv0ZcLnft2pXF3L3A3afcdb5o0aIsVoJ3hgEAANBYTIYBAADQWJNO\nhiPiKxGxOyLWt8S+GBHPR8Tasa8P9XeYAAAAQO+VvDN8q6QPmPifpZRWjH19t7fDAgAAAPpv0qqu\nlNJ9EbGk/0OZGm4Rtis6WL9+fRZzRXCu6GrOnDlZrLq4vM4FVtOFy+XmzZuz2GOPPZbFXKGA68I0\na9asLDbsuXTFES7mCsbmzp2bxaqFLS5HrvjFXW+uuKm0s5jbr1rQ4TqvuQ5trtDOnSuu4Kjfncsm\neh6Xs5kzZ2YxVzC2ffv2tu033ngj2+fxxx/PYu5ac4VArvjS5cwVT1W/1+XRFcu53LrXNdUd6A4c\nONB2jFatWpXt47qkutfjfh69973vbdt+3/vel+3jCs26udeVHq/qsXavaWRkJIs999xzWcydx2ed\ndVYWu+iii4rG1omUUtt57Ap5f/CDH2Qx15XzuOOOy2I/93M/N+G25O9N/SwAPajk/rpx48YstmPH\njizmiqyXLVuWxVx+S3RzNH47Ih4fW0Yxu4vHAQAAAAai08nwn0taKmmFpBcl/cl4O0bEzRExEhEj\nrvc86oNcDg9yORxa8+jeSUJ9tObSffwW6oPrsn46mgynlHallN5JKR2Q9JeSLp5g31tSSitTSivn\nzZvX6TgxDZDL4UEuh0NrHt0SFdRHay7d56eiPrgu66ejyXBELGzZ/GVJ+QJbAAAAYJqbtIAuIm6X\ndKWkuRGxQ9J/knRlRKyQlCRtk/QbfRxjx9yCfVe48f3vfz+LuWIL9w7a5ZdfnsVmzJhROkQUKu2K\n5jr5vPLKK1mMXB4aV7zgOi2uWLEii1VzVy3EknwuHVfU44pDSlW7T7rXWVpo4gr+3Pf2q+AyItoe\n2xWuuXepzjnnnCzmChW3bdvWtu0K2Vxhoet0dcwxx2Sx0kI+t6ynWjxVWqzl9ivNYz8LkN555522\nc9MVjLnj4F6P68i1cuXKtm13XnTTjcxxRYiuCLp6v3YdRN3PbFdod+KJJ2YxV/R+/vnnZ7FeOXDg\nQFunS1f46Irq3LFxnfLe8573tG27n22luSzlculi1YI517Vy9erVWcx1Gl28eHEWqxaCStJJJ52U\nxUqUfJrEJ034yx09GwAAADCN0IEOAAAAjcVkGAAAAI3FZBgAAACNNema4emqZNG+W4DuOl09/PDD\nWcwtLq8uVJekpUuXZrGp6OwyTEpy6Rbnu89vfOCBB4qes1pAIvnihGHrLlfCnb+uc5k792+44YYs\nVi18ePbZZ7N9XMGqK2RyBXSu8Mp1OnKFXM8880zbtis2WrBgQRZzH33ljtFkxVi9Pr9aH88Vn5UW\nFLn9nnzyybbt3bt3Z/u4e6477q6T1vHHH5/FXAc6VwRdzaMrhHTP6XLrxuGOZT/v82+99ZZ27dr1\n0+0tW7Zk+7iiU3e+LV++PItVr133+hx3Hy49h13e3Dl01113tW1/61vfyvbZunVrFnP3BnduX3/9\n9Vls/vz5WaxXDhw40FZY6orlXOGpy4nLZbWQ2Z37pTlyP4vdNe3OPVeo/tBDD7Vtf+Mb38j2Wbdu\nXRZz9wzXWe+KK67IYp0WVDNrAwAAQGMxGQYAAEBjMRkGAABAY027NcOlH+DtVNczufVma9euzWI7\nd+7MYm4dolt/5NYpYXydfhj+22+/ne2zcePGLPb8889nMZfLK6+8Mou5dUpN5NZCHn300VnMfbi7\nW6+1cOHCtu3WtZAHuTVzbv2jWw/nuPOg2mBDyu8Z7gPblyxZksVmz56dxVyDlqmuH2hdG+jG49bN\nuvPeraWtrld0TTfeeOONLObOHRdz6xrduVJduyzl+XDXvKsJqJ6b442t100LJlNt1OCaoDju+nBr\n2avnvVtj77jXXNJsQZI2bNiQxb7zne9ksWrjJLeu2J2zF198cRb72Mc+lsWWLVuWxUrXTHfiwIED\nbflzzUHcz0XXmMbNN6o5cddg6X3IrQ929Rw//OEPs9iqVauyWDW/mzZtyvZx56xbH3zjjTdmMXdN\nd9ooi3eGAQAA0FhMhgEAANBYTIYBAADQWEyGAQAA0FjTroCulFtwXo3t27cv2+exxx7LYq7QzhVW\n/OzP/mwWc0U+TWzU4HRTDFn9XlcUsHr16izmcumKot73vvdlsakukplOWl+nK7YojbnroVqc4opf\nWouFDnLFHK5Yx32vG5srkqk2WHDNNFzRmSvSmA7nSusY3Pnsjot7La6IrFq84woo3fVXcq+WfJGY\n28+dA9Vz6uSTT872qTZ/kXxxmTtGLrf9zHdEtOXFFaa6a81dMy+88EIWW7NmTdu2a4jhntPtV1rk\n6O7Xjz/+eBarFma6QjLXbOGzn/1sFrvwwguzmLv/9LvQdbJmOC6X7loqOdbuena5dD9TX3rppSzm\nPnTg0UcfzWKueVl1vK5YzuXyM5/5TBY799xzs5i7T3WaS94ZBgAAQGMxGQYAAEBjMRkGAABAY006\nGY6IUyJiVURsjIgNEfG5sficiPheRGwZ+zP/FHoAAABgGispoHtb0udTSmsi4t2SHo2I70n6VUn3\nppS+FBFfkPQFSb93KE/eTYGVHWilS9mLL76Y7eM6p7gF16effnoWc0UZU91hqimqRTJ79uzJ9nn2\n2WezmCtO+Jmf+ZksRi67V1poVL3OXQGLK/pwBR6ugMft54qx3HirRWGugK60W9p0U1r05e7DJbkt\nLVJ0RV0uZ65gyHUmdN0oq3k75ZRTsn1cwZ8rMpwOZsyYofnz5/90++yzz872efrpp7OYO14jIyNZ\nrHr8169fn+3jzhVXlO46EbpCr1deeaXoOZYuXdq27QqsbrrppixW7ZAo+eK7qS6UPuyww9o6IrpC\nTtdZznXGffDBB7NY9fpynf5c4ZrrhOfmTNu2bctiritgyQcRXH311dk+LpfufO93Lif96Z9SejGl\ntGbs7/skbZJ0sqQPS7ptbLfbJH2koxEAAAAAA3JIb4VFxBJJF0paLWlBSungfyN2Ssqb2Y9+z80R\nMRIRI+7dPdQHuRwe5HI4kMfh0ZrLl19+edDDQRfIZf0UT4YjYqakv5H0Oymltt/FpNHfs9k1Dyml\nW1JKK1NKK92vqVAf5HJ4kMvhQB6HR2su58yZM+jhoAvksn6KJsMRMUOjE+G/Sin97Vh4V0QsHPv3\nhZLyRSQAAADANDZpAV2Mrkb+sqRNKaU/bfmnb0u6SdKXxv68qy8jHIcr+qgu4HYLv91if7d4vbqI\nX8q7VUkUXR3Uy25zUp7L7du3Z/u4gozZs/MPNXGda1znqToURU2F0uNQmvPqNVLavc3F3nzzzaLn\ndB2dWotYDqoWYLjzwunmfJ9u51mnRXXuGJd2KnRFcO453XO4+/Cpp57atu26TrriPqe0Y14/83jE\nEUdo7ty5P92+/PLLs32eeeaZLLZu3bos5u6TjzzySNv2U089le3jiq5Kc+SutQsuuCCLnXnmmVms\n+lpdF7nWY3OQy29pkW8/HX744W1dLC+99NJsn82bN2exVatWZbG9e/dmsbvvvrttu5pbyRcBu8JW\nd126/Lqfs+edd14Wu+qqq9q2XQGd6/brzr1+57Lk0yQulfQrkp6IiIN9+X5fo5Pgv46IX5P0nKSP\n9WxUAAAAwBSYdDKcUnpA0njT73yaDwAAANQEv+MHAABAYzEZBgAAQGOVrBkeOLdo3y3+3r9/f9v2\njh07sn19fRF3AAAH/klEQVSqnc0ktXX6Oei0007LYt10oqq+hulWRDNILifVQqnnnnuu6PtOPPHE\nLOZy6YoCSpHLQ1PSuaw0H+4adEVvrQUrE31v9TxzHY163SmzrkryWHotuGInV8jsrmfXPbJ6L5g1\na1bR2NzPkdJ897trWWunxmuuuSbbx3VZdMWFTz75ZBYrOe9dPlyHxkWLFmUxVyznCseqhY/ueUsL\n46ZrMfthhx3Wdu9ZuXJlto8rcnSv2xXHVb/XnReuMM7dN9315grQXS4vu+yyLFbtBFnaRa6bXPat\nAx0AAAAwrJgMAwAAoLGYDAMAAKCxmAwDAACgsaa8gG6y4oTS7j8uVi2iaC1AOMgVX7hCAddNxT1e\naREJRVajOs2lK4hxhXFuv+XLl2ex6dqtaLoqzVuJXl8frgiq2sFwvOeoFnS4IrvSTmt1ve57WSBY\negzc8XOdCV0nKtfdrHrdu+u7m1xMdR4jou14uI56n/rUp7KYK85au3ZtFtu1a1fbtjsHXBHq4sWL\ns5i7v7qiutJrq3qsS499aSfFQeSy9XXOmzcv2+eGG27IYu64rl69Oott2bKlbfu1117L9nHH2XV+\nc4WPK1asyGILFizIYu5arRbHlRbGDeK+yTvDAAAAaCwmwwAAAGgsJsMAAABoLCbDAAAAaKwpL6Br\nXRjdTeGG61pS7b5z4403Zvt86EMfymKucMMVy1F01R/uGFaLZK677rpsn6uuuqrosdzCfpfzuhZA\nDUovC1a6KbxyhTmuU9YZZ5yRxaqduJYsWZLt47p6ufPH3ZOm2znV72K5bh6/NLeumKz6vK5IyeXR\n3dNL89hvrc/pzrc5c+Zksfe+971ZzBXVuaLTQx3TQS5vve4q1qluiu/6NQ6XS1es6PLmitmq3eVc\nbrvJWzcFxINABzoAAADgEDEZBgAAQGNNOhmOiFMiYlVEbIyIDRHxubH4FyPi+YhYO/aVrz8AAAAA\nprGSNcNvS/p8SmlNRLxb0qMR8b2xf/uzlNIf93JApWvs3Nq06vovt1bUrc3pxnRZJ1MXpceruq7K\nrbOainFgVL8/0N6tQXNr1Y499tgs5q5ztz7UrQeurpFzDR3c45c0CxgvNki9Xufb6XOWru1362Ld\nusZTTz21bds1Upo9e3bRc06Xtd+TPX7p87vXg6nVz1y6cxiHbtLJcErpRUkvjv19X0RskpS3cQMA\nAABq5pDWDEfEEkkXSjrYE/C3I+LxiPhKROT/7QYAAACmseLJcETMlPQ3kn4npfSapD+XtFTSCo2+\nc/wn43zfzRExEhEje/bs6cGQMSjkcniQy+FAHocHuRwe5LJ+iibDETFDoxPhv0op/a0kpZR2pZTe\nSSkdkPSXki5235tSuiWltDKltNJ99iPqg1wOD3I5HMjj8CCXw4Nc1s+ka4ZjdGX3lyVtSin9aUt8\n4dh6Ykn6ZUnrD/XJuylAmG6FKU1Xmo9eFl11Mw50r9NjXfp9pUV1rnGCa5pTcu51UyhV13Ov39df\nacwVB7nczpo1a9LHc+dOaayueQTQuZJPk7hU0q9IeiIi1o7Ffl/SJyNihaQkaZuk3+jLCAEAAIA+\nKfk0iQckuf8qf7f3wwEAAACmDh3oAAAA0FhMhgEAANBYJWuGgZ6hOAWl6tDRDZMrLZZzxWy9fE4A\nGA/vDAMAAKCxmAwDAACgsZgMAwAAoLGYDAMAAKCxwnVl6tuTReyR9NyUPWG7uZL2Dui5e6EX41+c\nUupJb0hy2ZVux9+zPEoDzWXd8yhNo1xyTXaNXI6qey75WfnPyGVhLqd0MjxIETGSUlo56HF0qu7j\n76W6H4u6j79XhuE4DMNr6IVhOA7D8Bp6oe7Hoe7j76W6H4upHD/LJAAAANBYTIYBAADQWE2aDN8y\n6AF0qe7j76W6H4u6j79XhuE4DMNr6IVhOA7D8Bp6oe7Hoe7j76W6H4spG39j1gwDAAAAVU16ZxgA\nAABoM5ST4Yj4SkTsjoj1LbE5EfG9iNgy9ufsQY5xIhFxSkSsioiNEbEhIj43Fq/Na+gF8jg8yOXw\nIJfDg1wOh7rnURp8LodyMizpVkkfqMS+IOnelNKZku4d256u3pb0+ZTSOZIukfSbEXGO6vUaeuFW\nkcdhcavI5bC4VeRyWNwqcjkMblW98ygNOpcppaH8krRE0vqW7c2SFo79faGkzYMe4yG8lrskXVvn\n10Aem51HcjlcX+RyeL7I5XB8DVMeB5HLYX1n2FmQUnpx7O87JS0Y5GBKRcQSSRdKWq2avoYeq+Ux\nII9WLY8DubRqeRzIpVXL40AuM7U9BoPIZZMmwz+VRv+LMe0/RiMiZkr6G0m/k1J6rfXf6vIa+qku\nx4A8Tq4ux4FcTq4ux4FcTq4ux4FcTqxOx2BQuWzSZHhXRCyUpLE/dw94PBOKiBkaPSH+KqX0t2Ph\nWr2GPqnVMSCPE6rVcSCXE6rVcSCXE6rVcSCX46rdMRhkLps0Gf62pJvG/n6TRtejTEsREZK+LGlT\nSulPW/6pNq+hj2pzDMjjpGpzHMjlpGpzHMjlpGpzHMjlhGp1DAaey0Evku7Hl6TbJb0o6S1JOyT9\nmqQTNFqJuEXSPZLmDHqcE4z//Rr9VcDjktaOfX2oTq+BPJJHcjmcX+RyeL7I5XB81T2P0yGXdKAD\nAABAYzVpmQQAAADQhskwAAAAGovJMAAAABqLyTAAAAAai8kwAAAAGovJMAAAABqLyTAAAAAai8kw\nAAAAGuv/A7waJ9gxyNuaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11f9cd438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "z1 = z[:,0]\n",
    "z2 = z[:,1]\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(z1,z2,'ko')\n",
    "plt.title(\"latent space\")\n",
    "\n",
    "#np.where((z1>3) & (z2<2) & (z2>0))\n",
    "#select the points from the latent space\n",
    "a_vec = [2,5,7,789,25,9993]\n",
    "for i in range(len(a_vec)):\n",
    "    ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro')  \n",
    "    ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]), \n",
    "                xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)\n",
    "\n",
    "\n",
    "f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6,  sharex='col', sharey='row',figsize=(12,2.5))\n",
    "for i in range(len(a_vec)):\n",
    "    eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "    eval('ax%d' %(i)).set_title('z%d'%i)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
